1820 lines
60 KiB
TeX
1820 lines
60 KiB
TeX
\documentclass{article}
|
||
|
||
\usepackage{url}
|
||
\usepackage{tex2page}
|
||
|
||
% typeset math as ascii
|
||
\htmlmathstyle{no-in-text-image no-display-image}
|
||
|
||
% something other than |
|
||
\verbescapechar\&
|
||
|
||
\cssblock
|
||
h1 {font-size: 16pt}
|
||
h2 {font-size: 15pt}
|
||
\endcssblock
|
||
|
||
% color of verbatim elements
|
||
\cssblock
|
||
.verbatim {color: grey20}
|
||
|
||
.scheme .variable {color: grey20}
|
||
.scheme .keyword {color: navy}
|
||
.scheme .builtin {color: maroon}
|
||
\endcssblock
|
||
|
||
% add some extra keywords
|
||
\scmkeyword{as case class data default deriving do else hiding if}
|
||
\scmkeyword{import in infix infixl infixr instance let module newtype}
|
||
\scmkeyword{of qualified then type where forall \\ }
|
||
\scmbuiltin{: :: = -> <- @ ~ => - >>= >> }
|
||
|
||
\newcommand{\code}[1]{{\texttt{#1}}}
|
||
\newcommand{\hsplugins}{{\texttt{hs-plugins}}}
|
||
|
||
\title{hs-plugins\\
|
||
Dynamically Loaded Haskell Modules}
|
||
|
||
\author{\urlh{http://www.cse.unsw.edu.au/~dons}{Don Stewart}}
|
||
|
||
\begin{document}
|
||
|
||
\maketitle
|
||
|
||
\medskip
|
||
%
|
||
{\htmlonly \textbf{Download \endhtmlonly
|
||
\urlh{ftp://ftp.cse.unsw.edu.au/pub/users/dons/hs-plugins/hs-plugins-0.9.8.tar.gz}
|
||
{version 0.9.8}}
|
||
%
|
||
\medskip
|
||
|
||
\hsplugins{} is a library for loading plugins written in Haskell into an
|
||
application at runtime. It also provides a mechanism for (re)compiling
|
||
Haskell source at runtime. Thirdly, a combination of runtime compilation
|
||
and dynamic loading provides a suite of \code{eval} functions. Values
|
||
exported by plugins are transparently available to Haskell host
|
||
applications, and bindings exist to use Haskell plugins from at least C
|
||
and Objective C programs. \hsplugins{} currently requires ghc-6.2.2.
|
||
|
||
\medskip
|
||
|
||
% grr. double spaced.
|
||
|
||
\tableofcontents
|
||
|
||
\newpage
|
||
|
||
\section{Download}
|
||
|
||
\begin{itemize}
|
||
|
||
\item
|
||
Download the latest stable release:\\
|
||
\url{ftp://ftp.cse.unsw.edu.au/pub/users/dons/hs-plugins/hs-plugins-0.9.8.tar.gz}
|
||
|
||
|
||
\item
|
||
Nightly cvs src snapshots are available at:\\
|
||
\url{ftp://ftp.cse.unsw.edu.au/pub/users/dons/hs-plugins/snapshots/}
|
||
|
||
|
||
\item
|
||
A tarball of the document you are reading:\\
|
||
\url{http://www.cse.unsw.edu.au/~dons/hs-plugins/hs-plugins.html.tar.gz}
|
||
|
||
|
||
\item
|
||
A postscript version of the document you are reading:\\
|
||
\url{http://www.cse.unsw.edu.au/~dons/hs-plugins/hs-plugins.ps.gz}
|
||
|
||
|
||
\item
|
||
A paper on interesting uses of \hsplugins{} to enable Haskell to be used
|
||
as an application extension language:\\
|
||
\url{http://www.cse.unsw.edu.au/~dons/hs-plugins/paper}
|
||
|
||
\end{itemize}
|
||
|
||
It is known to run on \code{i386-\{linux,freebsd,openbsd\}},
|
||
\code{ia64-linux}, \code{sparc-solaris2} and \code{powerpc-\{macosx,linux\}}.
|
||
|
||
\section{History}
|
||
|
||
\begin{itemize}
|
||
\item v0.9.8
|
||
\begin{itemize}
|
||
\item Fix bug in .hi parsing.
|
||
\item Add reloading of packages.
|
||
\item Fix bug in canonical module names
|
||
(fixing problems with "Foo.o" and "./Foo.o"
|
||
\item Fix for hierarchical names, don't guess them,
|
||
read them from the .hi file.
|
||
\item Add new varients of load.
|
||
\item Fix bug in makeAll, such that dependent module
|
||
changes were not noticed.
|
||
\item Add varient of eval:\code{ unsafeEval\_}, returing
|
||
Either.
|
||
\item Better, bigger testsuite.
|
||
\item Better api.
|
||
\end{itemize}
|
||
|
||
\item Septemeber 2004.
|
||
\begin{itemize}
|
||
\item makeAll
|
||
\item Better return type for make.
|
||
\end{itemize}
|
||
|
||
\item Mid August 2004, v0.9.6 release.
|
||
\begin{itemize}
|
||
\item More portable, thanks to debugging by Niklas Broberg.
|
||
\item Other small fixes to the interfaces.
|
||
\item Provides a runtime-generated printf.
|
||
\end{itemize}
|
||
|
||
\item Mid July 2004, added new pdynload strategy.
|
||
|
||
\item Mid-June 2004, v0.9.5 release.
|
||
\begin{itemize}
|
||
\item dynamic typing is working
|
||
|
||
\item static typing of interfaces is working
|
||
|
||
\item Adds \code{eval}, and \code{hs\_eval}
|
||
|
||
\item bugs fixed.
|
||
\end{itemize}
|
||
|
||
\item Early-June 2004, v0.9.4 release.
|
||
\begin{itemize}
|
||
|
||
\item Adds a .hi file parser. We use this to work out
|
||
plugin dependencies directly, meaning no more
|
||
\code{.dep} files or \code{ghcp}.
|
||
|
||
\item It also adds a package.conf parser, meaning we
|
||
can properly handle packages that either aren't stored
|
||
in the normal location, don't have a canonical name,
|
||
or are found using a -package-conf argument. Thanks to
|
||
Sean for this work.
|
||
|
||
\item the interface to load() has changed to allow a
|
||
list of package.conf files to search for packages.
|
||
|
||
\item the interace to make() has changed, so that you
|
||
can get back any stderr output produced during plugin
|
||
compilation.
|
||
|
||
\item It solves a bug whereby a package that is
|
||
required by another package would not be loaded unless
|
||
the plugin itself depended on this indirect package.
|
||
|
||
\item more stable, more examples.
|
||
\end{itemize}
|
||
|
||
\item May 2004, v0.9.3 released, adding support for dependency
|
||
conflict resolution between multiple plugins. Several plugins with
|
||
shared dependencies can now be safely loaded at once. --prefix is now
|
||
respected in ./configure. Thanks to Sean for this patch.
|
||
|
||
\item v0.9.2 change licence to LGPL
|
||
|
||
\item v0.9.1 expand on the documentation
|
||
|
||
\item v0.9 released, initial source release
|
||
|
||
\end{itemize}
|
||
|
||
\section{Acknowledgements}
|
||
|
||
\begin{itemize}
|
||
|
||
\item Andr\'e Pang's \code{runtime\_loader} was the inspiration and basis
|
||
of the dynamic loader (\url{http://www.algorithm.com.au}).
|
||
\hsplugins{} has benefited from many discussions with him,
|
||
particularly to do with dependency checking and dynamic typing, and
|
||
bug reports. Andr\'e wrote an objective C binding to hs-plugins, and
|
||
helped with the design of eval(). He also fixed GHC so we could load
|
||
the dynamic loader dynamically.
|
||
|
||
\item Sean Seefried (\url{http://www.cse.unsw.edu.au/~sseefried}) was
|
||
the first user of \hsplugins{} and his code and feedback have helped
|
||
make the library much more useful and powerful.
|
||
|
||
\item Manuel Chakravarty's \code{take} system provided the basis for
|
||
\code{make}, and helped with several issues to do with safety of
|
||
plugins, apis and the applications that use them.
|
||
Manuel also helped with the design of eval(), and on how to
|
||
successfully evaluate polymorphic functions using rank-N types.
|
||
|
||
\item Simon Marlow helped with several issues to do with linking and
|
||
loading static and dynamic code, and provided many useful suggestions.
|
||
|
||
\item Hampus Ram's dynamic loader
|
||
(\url{http://www.dtek.chalmers.se/~d00ram/dynamic/}) provided the
|
||
design of the state maintained by the loader, and for thread safety
|
||
issues relating to this.
|
||
|
||
\item Shae Erisson provided several insights into more powerful uses
|
||
of the library. Thanks to everyone on \#haskell who provided
|
||
discussion about the library.
|
||
|
||
\item Malcolm Wallace's \code{hmake} provided some useful insights in
|
||
how to compile Haskell source in a way that makes it appear like an
|
||
interpreter, used in the interactive environment: \code{plugs}.
|
||
|
||
\item Niklas Broberg helped a lot by testing, and providing feedback for
|
||
the new make and load API. Thanks Niklas.
|
||
|
||
\item Finally, thanks to everyone who has worked on GHC and its
|
||
libraries: for GHCi, the .hi interface parser, the package system, and
|
||
all the other code the \hsplugins{} depends on.
|
||
|
||
\end{itemize}
|
||
|
||
\newpage
|
||
|
||
\section{Overview}
|
||
|
||
\hsplugins{} is a library for compiling and loading Haskell code into a
|
||
program at runtime. It allows you to write a Haskell program (which may
|
||
be spread over multiple modules) and have an application (implemented in
|
||
any language with a Haskell FFI binding, including Haskell) load your
|
||
code at runtime, and use the values found within.
|
||
|
||
\hsplugins{} provides 3 major features:
|
||
%
|
||
\begin{itemize}
|
||
\item a dynamic loader,
|
||
\item a compilation manager, and
|
||
\item a Haskell evaluator
|
||
\end{itemize}
|
||
|
||
The \emph{dynamic loader} loads objects into the address space of an
|
||
application, along with any dependencies the plugin may have. The
|
||
loader is a binding to the GHC loader, which does single object
|
||
loading. GHC also performs the necessary linking of new objects into
|
||
the running process. On top of the GHC loader is our Haskell layer
|
||
that arranges for module and package dependencies to be found prior to
|
||
loading individual modules.
|
||
|
||
The \emph{compilation manager} is a \code{make}-like system for
|
||
compiling Haskell source code into a form suitable for loading
|
||
dynamically. While plugins are normally thought of as strictly object
|
||
code, there are a variety of scenarios where it is desirable to be
|
||
able to inspect the source code of a plugin, or to be able to
|
||
recompile a plugin at runtime. The compilation manager fills this
|
||
role. It is particularly useful in the implementation of \code{eval},
|
||
and \code{printf}.
|
||
|
||
The \emph{evaluator}, \code{eval}, is a client of the loader and
|
||
compilation manager. When passed a string of Haskell code, it compiles
|
||
the string to object code, loads the result, and returns a Haskell
|
||
value representing the compiled string to the caller. It can be
|
||
considered a Haskell interpreter, implemented as a library.
|
||
|
||
\section{Dynamic Loader}
|
||
|
||
The interface to the \hsplugins{} library can be divided into a number
|
||
of sections representing the functional units of the library.
|
||
Additionally, depending on the level of trust the application places
|
||
in the plugins, a variety of additional checks can be made on the
|
||
plugin as it is loaded. The levels of type safety possible are
|
||
summarised at the end of Section \ref{sec:compilation-manger} section.
|
||
The dynamic loader is available by using \code{-package plugins}.
|
||
|
||
\subsection*{Interface}
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
load :: FilePath
|
||
-> [FilePath]
|
||
-> [PackageConf]
|
||
-> Symbol
|
||
-> IO (LoadStatus a)
|
||
}
|
||
|
||
\scm{
|
||
load_ :: FilePath
|
||
-> [FilePath]
|
||
-> Symbol
|
||
-> IO (LoadStatus a)
|
||
}
|
||
|
||
\scm{
|
||
data LoadStatus a
|
||
= LoadSuccess Module a
|
||
| LoadFailure Errors
|
||
}
|
||
\end{quote}
|
||
%
|
||
Example:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
do mv <- load "Plugin.o" ["api"] [] "resource"
|
||
case mv of
|
||
LoadFailure msg -> print msg
|
||
LoadSuccess _ v -> return v
|
||
}
|
||
\end{quote}
|
||
%
|
||
This is the basic interface to the dynamic loader. Load the object file
|
||
specified by the first argument into the address space (the library will
|
||
preload any module or package dependencies). The second argument is an
|
||
include path to any additional objects to load (possibly the API of the
|
||
plugin). The third argument is a list of paths to any user-defined
|
||
\code{package.conf} files, specifying packages unknown to the GHC
|
||
package system. \code{Symbol} is a string specifying the symbol name you
|
||
wish to lookup. \code{load} returns a \code{LoadStatus} value representing
|
||
failure, or an abstract representation of the module (for calls to
|
||
\code{unload} or \code{reload}) with the symbol as a Haskell value. The
|
||
value returned must be given an explicit type signature, or provided
|
||
with appropriate type constraints such that GHC can determine the
|
||
expected type returned by \code{load}, as the return type is notionally
|
||
polymorphic.
|
||
|
||
\code{load\_} is provided for the common situation where no user-defined
|
||
package.conf files are required.
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
dynload :: Typeable a
|
||
=> FilePath
|
||
-> [FilePath]
|
||
-> [PackageConf]
|
||
-> Symbol
|
||
-> IO (LoadStatus a)
|
||
}
|
||
\end{quote}
|
||
%
|
||
Example:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
do mv <- dynload "Plugin.o" ["api"] ["plugins.conf.inplace"] "resource"
|
||
case mv of
|
||
LoadFailure msg -> print msg
|
||
LoadSuccess _ v -> putStrLn v
|
||
}
|
||
\end{quote}
|
||
%
|
||
\code{dynload} is a safer form of \code{load}. It uses dynamic types
|
||
to perform a check on the value returned by \code{load} at runtime, to
|
||
ensure that it has the type the application expects it to have.
|
||
\code{pdynload} is on average 7\% slower than an unchecked load.
|
||
|
||
In order to use \code{dynload}, the symbol the plugin exports must be
|
||
of type \code{AltData.Dynamic:Dynamic}. (See the \code{AltData} library
|
||
distributed with \hsplugins{}, and the \hsplugins{}
|
||
\code{examples/dynload} directory. References to \code{Typeable} and
|
||
\code{Dynamic} refer to the \hsplugins{} reimplementation of these
|
||
libraries. \code{AltData.Dynamic} is used at the moment, as there is a
|
||
limitation in the existing Data.Dynamic library in the presence of
|
||
dynamic loading. This will be fixed soon).
|
||
|
||
The value wrapped up in the \code{Dynamic} must be an instance of
|
||
\code{AltData.Typeable}. If the value exported by the plugin \emph{is}
|
||
of type \code{Dynamic}, and the value wrapped by the \code{Dynamic}
|
||
does not match the type expected of it by the application,
|
||
\code{dynload} will return \code{Nothing}, indicating that the plugin
|
||
is not typesafe with respect to the application. If the value passes
|
||
the typecheck, \code{dynload} will return \code{LoadSuccess}. If the value
|
||
exported by the plugin is \emph{not} of type \code{Dynamic},
|
||
\code{dynload} will crash---this is a limitation of the existing
|
||
\code{Dynamic} library, it can only type-check \code{Dynamic} values.
|
||
Additionally, Data.Dynamic is limited to monomorphic types, or must be
|
||
wrapped inside a rank-N type to hide the polymorphism from the
|
||
typechecker. This is a bit cumbersome. An alternative typesafe
|
||
\code{load} is available via the \code{pdynload} interface, which is
|
||
able to enforce the type of the plugin using GHC's type inference
|
||
mechanism, and is not restricted in its expressiveness (at the cost of greater load
|
||
times):
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
pdynload :: FilePath
|
||
-> [FilePath]
|
||
-> [PackageConf]
|
||
-> Type
|
||
-> Symbol
|
||
-> IO (LoadStatus a)
|
||
}
|
||
\end{quote}
|
||
%
|
||
Example:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
do v <- pdynload "Plugin.o" ["api"] [] "API.Interface" "resource"
|
||
case v of
|
||
LoadSuccess _ a -> putStrLn "yay!"
|
||
_ -> putStrLn "type error"
|
||
}
|
||
\end{quote}
|
||
%
|
||
\code{pdynload} is a replacement for \code{dynload}, which provides a
|
||
solution to the various problems caused by the existing dynamics
|
||
library in Haskell. Rather than use normal dynamics, which constrain
|
||
us to monomorphic types only (or rank-N types), it instead uses GHC's
|
||
type inference to unify the plugin's export value with that provided
|
||
by the api (via its .hi file). It is a form of \emph{staged type inference}
|
||
for module interfaces, allowing plugins to use any type definable in Haskell.
|
||
\code{pdynload} is like \code{dynload}, but requires a new \code{Type}
|
||
argument. This can be considered a type annotation on the value the plugin
|
||
should be constrained to.
|
||
|
||
The type of the plugin's resource field must be equivalent to the
|
||
\code{Type}. Prior to loading the object, \code{pdynload} generates a
|
||
tiny Haskell source file containing, for example:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
module APITypeConstraint where
|
||
import qualified API
|
||
import qualified Plugin
|
||
|
||
_ = Plugin.resource :: API.Interface
|
||
}
|
||
\end{quote}
|
||
%
|
||
It then calls GHC's type checker on this file, which runs the full
|
||
Haskell type inference machinery. If the file typecheckes, then the
|
||
plugin type is correct, and the plugin is safe to load, otherwise it
|
||
is an error.
|
||
|
||
Because we use the full Haskell type checker, we can have a form of
|
||
dynamic typechecking, on any type expressable in Haskell. A plugin's
|
||
value may, for example, have class constraints -- something not
|
||
checkable using the standard Dyanmic type. The cost is that
|
||
\code{pdynload} is roughly 46\% slower than an unchecked load.
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
unload :: Module -> IO ()
|
||
}
|
||
\end{quote}
|
||
|
||
Unload an object, \emph{but not its dependencies} from the address
|
||
space.
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
reload :: Module -> Symbol -> IO (LoadStatus a)
|
||
}
|
||
\end{quote}
|
||
|
||
Unload, and then reload a module that must have been previously
|
||
loaded. Doesn't reload the dependencies. \code{reload} is useful in
|
||
conjunction with \code{make}---a call to \code{reload} can be
|
||
performed if \code{make} has recompiled the plugin source.
|
||
|
||
Additionally, some support is provided to manipulation of
|
||
libraries of Haskell modules (usually known as packages):
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
loadPackage :: String -> IO ()
|
||
|
||
unloadPackage :: String -> IO ()
|
||
|
||
loadPackageWith :: String -> [PackageConf] -> IO ()
|
||
}
|
||
\end{quote}
|
||
|
||
\code{loadPackage} explcitly pulls in a library (which must be visible
|
||
in the current package namespace. \code{unloadPackage} unloads it.
|
||
\code{loadPackageWith} behaves like \code{loadPackage}, but you are able
|
||
to supply extra package.confs to augment the library search path.
|
||
|
||
Examples:
|
||
\begin{quote}
|
||
\scm{
|
||
do loadPackageWith "yi" ["yi.conf"]
|
||
unloadPackage "yi"
|
||
}
|
||
\end{quote}
|
||
|
||
\newpage
|
||
|
||
\section{Compilation Manager}
|
||
|
||
The compilation manager is the system by which Haskell source code is
|
||
compiled to object code suitable for loading.
|
||
|
||
\subsection*{Interface}
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
make :: FilePath
|
||
-> [Arg]
|
||
-> IO MakeStatus
|
||
|
||
makeAll :: FilePath
|
||
-> [Arg]
|
||
-> IO MakeStatus
|
||
|
||
data MakeStatus
|
||
= MakeSuccess MakeCode FilePath
|
||
| MakeFailure Errors
|
||
|
||
data MakeCode = ReComp | NotReq
|
||
}
|
||
\end{quote}
|
||
|
||
Compile a Haskell source file to an object file, with any arguments
|
||
specified in the argument list passed through to GHC. Returns the
|
||
build status.
|
||
|
||
\code{make} generates a GHC \code{.hi} file containing a list of
|
||
package and objects that the source depends on. Subsequent calls to
|
||
\code{load} will use this interface file to load module and library
|
||
dependencies prior to loading the object itself. \code{makeAll} also
|
||
recursively compiles any dependencies it can find using GHC's
|
||
\code{--make} flag.
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
merge :: FilePath -> FilePath -> IO MergeStatus
|
||
|
||
mergeTo :: FilePath -> FilePath -> FilePath -> IO MergeStatus
|
||
|
||
data MergeStatus
|
||
= MergeSuccess MergeCode Args FilePath
|
||
| MergeFailure Errors
|
||
|
||
type MergeCode = MakeCode
|
||
}
|
||
\end{quote}
|
||
|
||
The merging operation is extremely useful for providing extra default
|
||
syntax. An EDSL user then need not worry about declaring module names,
|
||
or having required imports. In this way, the stub file can also be
|
||
used to provide syntax declarations that would be inconvenient to
|
||
require of the plugin author. \code{merge} will include any import and
|
||
export declarations written in the stub, as well as any module name,
|
||
so that plugin author's need not worry about this compulsory syntax.
|
||
Additionally, if a plugin requires some non-standard library, which
|
||
must be provided as a \code{-package} flag to GHC, they may specify
|
||
this using the non-standard \code{GLOBALOPTIONS} pragma. Options
|
||
specified in the source this way will be added to the command line.
|
||
This is useful for users who wish to use GHC flags that cannot be
|
||
specified using the conventional \code{OPTIONS} pragma.
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
makeWith :: FilePath
|
||
-> FilePath
|
||
-> [Arg]
|
||
-> IO MakeStatus
|
||
}
|
||
\end{quote}
|
||
|
||
This is a variety of \code{make} that first calls \code{merge} to
|
||
combine the plugin source with a syntax stub. The result is then
|
||
compiled. This is the preferred interface to EDSL authors who wish to
|
||
add extra syntax to a user's source. It is important to note that the
|
||
module and types from the second file argument are used to override
|
||
any of those that appear in the first argument. For example, consider
|
||
the following source files:
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
module A where
|
||
|
||
a :: Integer
|
||
a = 1
|
||
}
|
||
\end{quote}
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
module B where
|
||
|
||
a :: Int
|
||
}
|
||
\end{quote}
|
||
|
||
Calling \code{makeWith "A" "B" []} will merge the module name and types
|
||
from module B into module A, generating a third file:
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
{-# LINE 1 "A.hs" #-}
|
||
module MxYz123 where
|
||
|
||
{-# LINE 3 "B.hs" #-}
|
||
a :: Int
|
||
{-# LINE 4 "A.hs" #-}
|
||
a = 1
|
||
}
|
||
\end{quote}
|
||
|
||
Leading to the desired result that we can ignore user-supplied module
|
||
names and types. Knowing the module name, in particular, is important
|
||
for dynamic loading, which requires the module name be known when
|
||
searching for symbols.
|
||
|
||
\subsection*{Levels of Safety}
|
||
|
||
The normal dynamic loader, using \code{load} on object files only,
|
||
places full trust in the author of the plugin to provide a type-safe
|
||
object file, containing valid code. This can be mitigated somewhat via
|
||
the use of \code{make} to ensure that the plugin is at least Haskell
|
||
code that is well-typed internally (if we trust GHC to compile it
|
||
correctly).
|
||
|
||
If we trust the user to provide an interface of \code{Dynamic} type, we
|
||
can check the plugin type at runtime, but the plugin's value must be
|
||
\code{Typeable}, which restricts it to be a monomorphic type (or to
|
||
using rank-N tricks).
|
||
|
||
The greatest safety can be obtained by using \code{pdynload}, at the
|
||
cost of increased load times. \code{pdynload} essentially performs full
|
||
type inference on the plugin interface at runtime. The type safety of
|
||
the plugin, using \code{pdynload}, is then as safe as if the plugin was
|
||
statically compiled into the application. It does not provide any
|
||
\emph{further} safety than exists in static compilation. For example, it
|
||
does not preclude the use of (evil) \code{unsafeCoerce\#} to defeat
|
||
type-checking, either statically or at runtime. An extensive discussion
|
||
of type safe plugin loading is available in the \hsplugins{} paper
|
||
\urlh{http://www.cse.unsw.edu.au/~dons/hs-plugins/paper}{here}.
|
||
|
||
\newpage
|
||
|
||
\section{Eval.Haskell}
|
||
|
||
\code{eval}, and its siblings, provide a mechanism to compile and run
|
||
Haskell code at runtime, in the form of a String. It is provided as a
|
||
separate package to the plugins package, and needs to be linked
|
||
against using \code{-package eval}. The general framework is that the
|
||
string is used to create a plugin source file, which is compiled and
|
||
loaded, and type checked against its use. The resulting value is
|
||
returned to the caller. It resembles the \code{eval} primitives of
|
||
scripting languages.
|
||
|
||
\subsection*{Interface}
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
eval :: Typeable a => String -> [Import] -> IO (Maybe a)
|
||
}
|
||
\end{quote}
|
||
|
||
\code{eval} takes a string, and a list of import module names, and
|
||
returns a \code{Maybe} value. \code{Nothing} means the code did not
|
||
compile. \code{Just v} gives you \code{v}, the result of evaluating
|
||
your code. It is interesting to note that \code{eval} has the type of
|
||
an interpreter. The \code{Typeable} constraint is used to type check
|
||
the evaluated code when it is loaded, using \code{dynload}. The
|
||
existing \code{Data.Dynamic} library requires that only monomorphic
|
||
values are \code{Typeable}, so in order to evaluate polymorphic
|
||
functions you need to wrap them up using rank-N types. Some
|
||
examples:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
import Eval.Haskell
|
||
|
||
main = do i <- eval "1 + 6 :: Int" [] :: IO (Maybe Int)
|
||
if isJust i then putStrLn (show (fromJust i)) else return ()
|
||
}
|
||
\end{quote}
|
||
|
||
When executed this program calls \code{eval} to compile and load the
|
||
simple arithmetic expression, returning the result, which is
|
||
displayed. If the value loaded is not of type \code{Int},
|
||
\code{dynload} will throw an exception.
|
||
|
||
The following example, due to Manuel Chakravarty, shows how to
|
||
evaluate a polymorphic function. Polymorphic values are not easily
|
||
made dynamically typeable, but this example shows how to do it. The
|
||
module \code{Poly} is imported as the second argument, providing the
|
||
type of the polymorphic function:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
import Poly
|
||
import Eval.Haskell
|
||
|
||
main = do m_f <- eval "Fn (\\x y -> x == y)" ["Poly"]
|
||
when (isJust m_f) $ do
|
||
let (Fn f) = fromJust m_f
|
||
putStrLn $ show (f True True)
|
||
putStrLn $ show (f 1 2)
|
||
}
|
||
\end{quote}
|
||
%
|
||
And the type of \code{Fn}:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
{-# OPTIONS -fglasgow-exts #-}
|
||
module Poly where
|
||
|
||
import AltData.Typeable
|
||
|
||
data Fn = Fn {fn :: forall t. Eq t => t -> t -> Bool}
|
||
|
||
instance Typeable Fn where
|
||
typeOf _ = mkAppTy (mkTyCon "Poly.Fn") []
|
||
}
|
||
\end{quote}
|
||
%
|
||
When executed, this program produces:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
$ ./a.out
|
||
True
|
||
False
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
We thus get dynamically typeable polymorphic functions.
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
unsafeEval :: String -> [Import] -> IO (Maybe a)
|
||
|
||
unsafeEval_ :: String
|
||
-> [Import]
|
||
-> [String]
|
||
-> [FilePath]
|
||
-> IO (Either [String] a)
|
||
}
|
||
\end{quote}
|
||
|
||
Wrapping up polymorphic values can be annoying, so we provide a
|
||
\code{unsafeEval} function for people who like to live on the edge,
|
||
which dispenses with dynamic typing, relying instead on the
|
||
application to provide the correct type annotation on the call to
|
||
\code{eval}. If the type loaded by \code{eval} is wrong,
|
||
\code{unsafeEval} will crash. However, its lets us remove some
|
||
restrictions on what types can be evaluated, which can be useful.
|
||
|
||
{unsafeEval\_} lets the application have full control over the import
|
||
environment and load flags to the eval call, which is useful for
|
||
applications that wish to script themselves, and require specific
|
||
modules and packages to be in scope in the eval-generated module.
|
||
|
||
This example maps a \code{toUpper} over a list:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
import Eval.Haskell
|
||
|
||
main = do s <- unsafeEval "map toUpper \"haskell\"" ["Data.Char"]
|
||
when (isJust s) $ putStrLn (fromJust s)
|
||
}
|
||
\end{quote}
|
||
|
||
And here we evaluate a lambda abstraction, applying the result to
|
||
construct a tuple. Note the type information that must be supplied in
|
||
order for Haskell to type the usage of \code{fn}:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
import Eval.Haskell
|
||
|
||
main = do fn <- unsafeEval "(\\(x::Int) -> (x,x))" [] :: IO (Maybe (Int -> (Int,Int)))
|
||
when (isJust fn) $ putStrLn $ show $ (fromJust fn) 7
|
||
}
|
||
\end{quote}
|
||
\subsection{Utilities for use with eval}
|
||
\code{hs-plugins} proves the following utilities for use with \code{eval}:
|
||
\begin{itemize}
|
||
\item
|
||
\code{mkHsValues} is a helper function for converting \code{Data.Map}s
|
||
of names and values into Haskell code. It relies on the assumption
|
||
that the passed values' Show instances produce valid Haskell
|
||
literals (this is true for all prelude types). It's type is as follows:
|
||
\begin{quote}
|
||
\scm{
|
||
mkHsValues :: (Show a) => Data.Map String a -> String
|
||
}
|
||
\end{quote}
|
||
\end{itemize}
|
||
\subsection{Foreign Eval}
|
||
|
||
A preliminary binding to \code{eval} has been implemented to allow C
|
||
(and Objective C) programs access to the evaluator. Foreign bindings
|
||
to the compilation manager and dynamic loader are yet to be
|
||
implemented, but shouldn't be too hard. An foreign binding to a
|
||
Haskell module that wraps up calls to \code{make} and \code{load}
|
||
would be fairly trivial.
|
||
|
||
At the moment we have an ad-hoc binding to \code{eval}, so that C
|
||
programmers who know the type of value that will be returned by
|
||
Haskell can call the appropriate hook into the evaluator. If they get
|
||
the type wrong, a nullPtr will be returned (so calling Haskell is
|
||
still typesafe). The foreign bindings to \code{eval} all return
|
||
\code{NULL} if an error occurred, otherwise a pointer to the value is
|
||
returned.
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
foreign export ccall hs_eval_b :: CString -> IO (Ptr CInt)
|
||
|
||
foreign export ccall hs_eval_c :: CString -> IO (Ptr CChar)
|
||
|
||
foreign export ccall hs_eval_i :: CString -> IO (Ptr CInt)
|
||
|
||
foreign export ccall hs_eval_s :: CString -> IO CString
|
||
}
|
||
\end{quote}
|
||
|
||
An example C program for compiling and evaluating Haskell code at
|
||
runtime follows. This program calculates a fibonacci number, returning
|
||
it as a \code{CString} to the C program:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
#include "EvalHaskell.h"
|
||
#include <stdio.h>
|
||
|
||
int main(int argc, char *argv[])
|
||
{
|
||
char *p;
|
||
hs_init(&argc, &argv);
|
||
p = hs_eval_s("show $ let fibs = 1:1:zipWith (+) fibs (tail fibs) in fibs !! 20");
|
||
if (p != NULL)
|
||
printf("%s\n",p);
|
||
else
|
||
printf("error in code\n");
|
||
hs_exit();
|
||
return 0;
|
||
}
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
\subsection{Eval.Printf}
|
||
|
||
It has been noted that \code{printf} format strings are the concrete syntax
|
||
of a string formatting interpreter (over 1000 lines long in libc!). By
|
||
combining runtime generation of new Haskell code, with dynamic typing,
|
||
it becomes possible to implement a typesafe \code{printf} for Haskell.
|
||
|
||
This has already been achieved in at least 3 different ways. A standard
|
||
solution (Hinze, Danvey) begins by supplying printf with the abstract
|
||
syntax of the formatting string, resolving the issue of the lack of
|
||
typing in the raw fmt string. An alternative solution (see Ian Lynagh's
|
||
Printf library) uses Template Haskell to transform a printf format
|
||
string into a new Haskell function at compile time, however this
|
||
requires that the format string is known at compile time. By using
|
||
runtime compilation we can take a similar approach, but instead generate
|
||
the print function at runtime! To make this safe, we then need to use
|
||
dynamic typing to check the newly-generated print function against its
|
||
arguments.
|
||
|
||
\subsection*{Printf Interface}
|
||
|
||
The \code{Printf} library implements a reasonable amount of the C
|
||
printf's functionality.
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
printf :: String -> [Dynamic] -> IO ()
|
||
}
|
||
\end{quote}
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
sprintf :: String -> [Dynamic] -> IO String
|
||
}
|
||
\end{quote}
|
||
|
||
Because the arguments to printf are of differing types, and the number
|
||
of arguments is not known at compile time, we simulate variadic
|
||
functions by using a heterogenous list of arguments. A special list
|
||
constructor, \code{!}, is provided for this. An example, noting the
|
||
syntax for constructing a heterogenous argument list:
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
import Eval.Printf
|
||
|
||
main = do printf "%d\n" $ (42::Int) ! []
|
||
printf "0x%X\n" $ (42::Int) ! []
|
||
printf "%f\n" $ (42.1234 :: Double) ! []
|
||
printf "%c:%c:%c\n" $ 'a' ! 'b' ! 'c' ! []
|
||
printf "%s\n" $ "haskell" ! []
|
||
printf "%010.4f\n" $ (42.1234 :: Double) ! []
|
||
printf "%10.4s\n" $ "haskell" ! []
|
||
printf "%-10.4s\n" $ "haskell" ! []
|
||
}
|
||
\end{quote}
|
||
|
||
Compiling this program against \code{-package eval}, and running it
|
||
produces the following output:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
42
|
||
0x2A
|
||
42.123400
|
||
a:b:c
|
||
haskell
|
||
00042.1234
|
||
hask
|
||
hask
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
If you mismatch the types specified in the format string, and the
|
||
types you apply printf to, printf will generate an exception, like so:
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
import Eval.Printf
|
||
|
||
main = printf "%d\n" ("badstring" ! [])
|
||
}
|
||
\end{quote}
|
||
|
||
The above code will generate this error, indicating that you attempted
|
||
to apply a string to a function that expected an Int:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
paprika$ ./a.out
|
||
Fail: Type error in dynamic application.
|
||
Can't apply function <Int -> [Char]> to argument <[Char]>
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
Note that this isn't the fastest printf implementation in the world. A
|
||
call to printf invokes GHC to transform the printf format string into
|
||
a Haskell code fragment, which is compiled and dynamically linked back
|
||
into the application, and then applied to its arguments. If you need
|
||
to use the same printf function against multiple times, you can save
|
||
recompilation, in which case printf runs as fast as other native code.
|
||
|
||
Additionally, it only implements the most common modifiers to the
|
||
basic conversion specifiers, and they have not all been fully tested.
|
||
|
||
\section{RTS Binding}
|
||
|
||
The low level interface is the binding to GHC's Linker.c. Therefore,
|
||
\hsplugins{} only works on platforms with a working GHCi. This library
|
||
is based on code from Andr<64> Pang's runtime loader. The low level
|
||
interface is as follows:
|
||
|
||
\begin{itemize}
|
||
\item \code{initLinker} \em start the linker up
|
||
\item \code{loadObject} \em load a vanilla .o
|
||
\item \code{loadPackage} \em load a GHC library and its cbits
|
||
\item \code{loadShared } \em load a .so object file
|
||
\item \code{resolveObjs} \em and resolve symbols
|
||
\end{itemize}
|
||
|
||
Additionally, \code{Hi.Parser} provides an interface to a GHC
|
||
\code{.hi} file parser. Currently we only parse just the dependency
|
||
information, import and export information from \code{.hi} files, but
|
||
all the code is there for an application to extract other information
|
||
from \code{.hi} files.
|
||
|
||
\newpage
|
||
|
||
\section{Dynamic Loader Implementation}
|
||
|
||
The dynamic loader is the system by which modules, and their
|
||
dependencies can be loaded, unloaded or reloaded at runtime, and
|
||
through which we access the functions we need.
|
||
|
||
At its lowest level, the \hsplugins{} loader is a binding to the GHC
|
||
runtime loader and linker. This layer is a direct reimplementation of
|
||
Andre Pang's \code{runtime\_loader} (barely any code changed). The
|
||
code at this level can only load single modules, or packages/archives
|
||
(which are just objects too). Any dependency resolution must be
|
||
performed by hand.
|
||
|
||
On top of Andre's interface is a more convenient interface through
|
||
which user's should interact with the dynamic loader. The most
|
||
significant extension to Andre's work is the automatic calculation and
|
||
loading of a plugin's package or module dependencies via .hi file
|
||
information. It also handles initialisation of the loader, and
|
||
retrieval of values from the plugin in a more convenient way. Some
|
||
state is also stored in the loader to keep track of which modules and
|
||
packages have been loaded, to prevent unnecessary (actually, fatal)
|
||
loading of object files and packages already loaded. Thus you can
|
||
safely load several plugins at once, that share common dependencies,
|
||
without worrying about the dependencies being loaded multiple times.
|
||
We also store package.conf information in the state, so we can work
|
||
out where a package lives and what it depends on.
|
||
|
||
The ability to remember which packages and objects have been loaded is
|
||
based on ideas in Hampus Ram's dynamic loader, which has a more
|
||
advanced dependency tracking system, with the ability to unload the
|
||
dependencies of a plugin. \hsplugins{} doesn't provide ``cascading
|
||
unloading''. The advantage \hsplugins{} has over Hampus' loader seems
|
||
to be the automatic dependency resolution via vanilla .hi files and
|
||
the dynamic recompilation stuff.
|
||
|
||
Using \code{load}, any library packages, or any \code{.o} files, that a
|
||
plugin depends upon will be automatically loaded prior to loading the
|
||
module itself. \code{load} then looks up a symbol from the object file,
|
||
and returns the value associated with the symbol as a conventional
|
||
Haskell value. It should also be possible to load a GHCi-style \code{.o}
|
||
archive of object files this way, although there is currently no way
|
||
to extract multple plugin interfaces from a archive of objects.
|
||
|
||
The application writer is not required to recalculate dependencies if
|
||
the plugin changes, and the plugin author does not need to specify
|
||
what dependencies exist, as is required in the lower level interface.
|
||
This is achieved by using the dependency information calculated by GHC
|
||
itself, stored in .hi files, to work out which modules and packages to
|
||
load, and in what order. A plugin in \hsplugins{} is really a pair of
|
||
an object file (or archive) and a \code{.hi} file, containing package
|
||
and module dependency information.
|
||
|
||
The \code{.hi} file is created by GHC when the plugin is compiled,
|
||
either by hand or via \code{make}. \code{load} uses a binary parser to
|
||
extract the relevant information from the \code{.hi} data. Because the
|
||
dependency information is stored in a separate file to the application
|
||
that loads the plugin, such information can be recalculated without
|
||
having to modify the application. Becaues of this, it was easy to
|
||
extend the load to support recompilation of module source, even if
|
||
dependencies change, because dependencies are no longer hard-coded
|
||
into the application source itself, but are specified by the plugin.
|
||
|
||
Assuming we have a plugin exporting some data, ``resource'', with a
|
||
record name \code{field :: String}, here is an example call to \code{load}:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
do m_v <- load "Test.o" ["."] [] "resource"
|
||
v <- case m_v of
|
||
LoadSuccess _ v -> return v
|
||
_ -> error "load failed"
|
||
putStrLn $ field v
|
||
}
|
||
\end{quote}
|
||
|
||
This loads the object file \code{Test.o}, and any packages or objects
|
||
\code{Test.o} depends on. It resolves undefined symbols, and returns
|
||
from the object file the Haskell value named ``resource'', as the
|
||
value ``v''. This must be a value exported by the plugin. We then
|
||
retrieve the \code{field} component of \code{v}, and print it out.
|
||
|
||
This simple usage assumes that the plugin to load is in the same
|
||
directory as the application, and that the api defining the interface
|
||
between plugin and application is also in the current directory (hence
|
||
the ``.'' in the 2nd argument to \code{load}).
|
||
|
||
\subsection*{Dynamically Loading the Dynamic Loader}
|
||
|
||
It is also possible to load the \code{plugins} or \code{eval}
|
||
libraries in GHC. A couple of recent patches to the linker have made
|
||
this possible (available in ghc-6.2.2 or in the head branch). Here,
|
||
for example, we load the \code{plugs} interactive environment in GHCi,
|
||
and evaluated some code. The source to \code{plugs} is in Appendix
|
||
\ref{sec:plugs}.
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
paprika$ ghci -package-conf ../../../plugins.conf.inplace -package eval
|
||
___ ___ _
|
||
/ _ \ /\ /\/ __(_)
|
||
/ /_\// /_/ / / | | GHC Interactive, version 6.3, for Haskell 98.
|
||
/ /_\\/ __ / /___| | http://www.haskell.org/ghc/
|
||
\____/\/ /_/\____/|_| Type :? for help.
|
||
|
||
Loading package base ... linking ... done.
|
||
Loading package altdata ... linking ... done.
|
||
Loading package unix ... linking ... done.
|
||
Loading package mtl ... linking ... done.
|
||
Loading package lang ... linking ... done.
|
||
Loading package posix ... linking ... done.
|
||
Loading package haskell98 ... linking ... done.
|
||
Loading package haskell-src ... linking ... done.
|
||
Loading package plugins ... linking ... done.
|
||
Loading package eval ... linking ... done.
|
||
Prelude> :l Main
|
||
Skipping Main ( Main.hs, Main.o )
|
||
Ok, modules loaded: Main.
|
||
Prelude Main> main
|
||
Loading package readline ... linking ... done.
|
||
__
|
||
____ / /_ ______ ______
|
||
/ __ \/ / / / / __ `/ ___/ PLugin User's GHCi System, for Haskell 98
|
||
/ /_/ / / /_/ / /_/ (__ ) http://www.cse.unsw.edu.au/~dons/hs-plugins
|
||
/ .___/_/\__,_/\__, /____/ Type :? for help
|
||
/_/ /____/
|
||
|
||
Loading package base ... linking ... done
|
||
plugs> map (\x -> x + 1) [0..10]
|
||
[1,2,3,4,5,6,7,8,9,10,11]
|
||
plugs> :t "haskell"
|
||
"haskell" :: [Char]
|
||
plugs> :q
|
||
*** Exception: exit: ExitSuccess
|
||
Prelude Main> :q
|
||
Leaving GHCi.
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
\subsection*{Dynamic Typing}
|
||
|
||
Support is also provided to unwrap and check the type of dynamically
|
||
typed plugin values (those wrapper in a \code{toDyn}) via
|
||
\code{dynload}. This is the same as \code{load}, except that instead
|
||
of a returning the value it finds, it unwraps a dynamically typed
|
||
value, checks the type, and returns the unwrapped value. This is to
|
||
provide further trust that the symbol you are retrieving from the
|
||
plugin is of the type you think it is, beyond that trust you have by
|
||
knowing that the plugin was compiled against a shared API. By using
|
||
\code{dynload} it is not enough for an object file to just have the
|
||
same symbol name as the function you require, it must also carry the
|
||
\code{Data.Dynamic} representation of the type, too. \code{pdynload}
|
||
rectifies most of \code{dynload}'s limitations, but at the cost of
|
||
additional running time.
|
||
|
||
\section{Compilation Manager Implementation}
|
||
|
||
Along side the dynamic loader is the compilation manager. This is a
|
||
\code{make}-like system for compiling Haskell source, prior to loading
|
||
it. \code{make} checks if a source file is newer than its associated
|
||
object file. If so, the source is recompiled to an object file, and a
|
||
new dependency file is created, in case the dependencies have changed
|
||
in the source. This module can then be loaded. The idea is to allow
|
||
EDSL authors to write plugins without having to touch a compiler: it
|
||
is all transparent. It also allows us to enforce type safety in the
|
||
plugin by injecting type constraints into the plugin source, as has
|
||
been discussed eariler.
|
||
|
||
The effect is much like \emph{hi} (Hmake Interactive), funnily enough.
|
||
An application using both \code{make} and \code{load} behaves like a
|
||
Haskell interpreter, using \code{eval}. You modify your plugin, and
|
||
the application notices the change, recompiles it (possibly issuing
|
||
type errors) and then reloads the object file, providing the
|
||
application with the latest version of the code.
|
||
|
||
An example:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
do status <- make "Plugin.hs" []
|
||
obj <- case status of
|
||
MakeSuccess _ o -> return o
|
||
MakeFailure e -> mapM_ putStrLn e >> error "failed"
|
||
|
||
m_v <- load obj ["api"] [] "resource"
|
||
v <- case m_v of
|
||
LoadSuccess _ v -> return v
|
||
_ -> error "load failed"
|
||
putStrLn $ field v
|
||
}
|
||
\end{quote}
|
||
|
||
\code{make} accepts a source file as an argument, and a (usually empty)
|
||
list of GHC flags needed to compile the object file. It then checks to
|
||
see if compilation is required, and if so, it calls GHC to compile the
|
||
code, with and arguments supplied. If any errors were generated by
|
||
GHC, they are returned as the third component of the triple.
|
||
|
||
Usually it will be necessary to ensure that GHC can find the plugin
|
||
API to compile against. This can be done by either making sure the API
|
||
is in the same directory as the plugin, or by adding a \code{-i} flag to
|
||
\code{make}'s arguments. If the API is created as a ``package'' with a
|
||
package.conf file, \code{make} can be given \code{-package-conf} arguments
|
||
to the same effect.
|
||
|
||
Normally, \code{make} generates the \code{.o} and \code{.hi} files in
|
||
the same directory as the source file. This is not always desirable,
|
||
particularly for interpreter-like applications. To solve this, you can
|
||
pass \code{[''-odir'', path]} as elements of the arg list to
|
||
\code{make}, and it will respect these arguments, generating the
|
||
object and interface file in the directory specified. GHC's argument
|
||
\code{''-o''} is also respected in a similar manner, so you could also
|
||
say \code{[''-o'', obj]} for the same effect.
|
||
|
||
\code{make} is entirely optional. All user's have to do to use the
|
||
loader on its own is make sure they only load object files that also
|
||
have a \code{.hi} file. This will usually be the case if the plugin is
|
||
compiled with GHC.
|
||
|
||
\subsection*{makeWith}
|
||
|
||
\code{makeWith} merges two source files together, using the function
|
||
and value declarations from one file, with any syntax in the second,
|
||
creating a new third source file. It then compiles this source file
|
||
via \code{make}.
|
||
|
||
This function exists as a benefit to EDSL authors and is related to
|
||
the original motivation for \hsplugins{}, as a .conf file language
|
||
library. Configuration files need to be clean and simple, and you
|
||
can't rely, or trust, the user to get all the compulsory details
|
||
correct. So the solution is to factor out any compulsory syntax, such
|
||
as module names, imports, and also to provide a default instance of
|
||
the API, and store this code in a separate file provided by the
|
||
application writer, not the user. \code{makeWith} then merges
|
||
whatever the user has written, with the syntax stub, generating a
|
||
complete Haskell plugin source, with the correct module names and
|
||
import declarations. We also ensure the plugin only exports a single
|
||
interface value while we are here.
|
||
|
||
\code{makeWith} thus requires a Haskell parser to parse two source files
|
||
and merge the results. We are merging abstract syntax here. This is
|
||
implemented using the Language.Haskell parser library. Unfortunately,
|
||
this library doesn't implement all of GHC's extensions, so if you wish
|
||
to use \code{makeWith} you can only write Haskell source that can be
|
||
parsed by this library, which is just H98 and a few extensions. This
|
||
is another short coming in the current design that will be overcome
|
||
with \code{-package ghc}. Remember, however, for normal uses of
|
||
\code{make} and \code{load} you are unrestricted in what Haskell you use.
|
||
This is the same restriction present in happy, the Haskell parser,
|
||
placed on the code you can provide in the \code{.y} source.
|
||
|
||
\code{makeWith} also makes use of line pragmas. If the merged file
|
||
fails to compile, the judicious use of line number pragmas ensure that
|
||
the user receives errors messages reported with reference to their
|
||
source file, and not line number in the merged file. This is a
|
||
property of the Language.Haskell parser that we can make use of.
|
||
|
||
An example of \code{makeWith}:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
do status <- makeWith "Plugin.in" "Plugin.stub" []
|
||
obj <- case status of
|
||
MakeFailure e -> mapM_ putStrLn e >> error "failed"
|
||
MakeSuccess _ o -> return o
|
||
m_v <- load obj [apipath] [] "resource"
|
||
v <- case m_v of
|
||
LoadSuccess _ v -> return v
|
||
_ -> error "load failed"
|
||
putStrLn $ field v
|
||
}
|
||
\end{quote}
|
||
|
||
We combine the user's file (\code{Plugin.in}) with a stub of syntax
|
||
generating a new, third Haskell file in the default tmpdir. This is
|
||
compiled as per usual, producing object and interface files. The
|
||
object is then loaded, and we extract the value exported.
|
||
|
||
Using \code{makeWith} it is possible to write very simple, clear
|
||
Haskell plugins, that appear not to be Haskell at all. It is an easy
|
||
way to get EDSL user's writing plugins that are actually Haskell
|
||
programs, for .e.g, configuration files. See the examples that come
|
||
with the src.
|
||
|
||
\newpage
|
||
|
||
\section{An Example}
|
||
|
||
This is an introductory example.
|
||
|
||
\subsection*{API}
|
||
|
||
First we need an interface between the application and the plugin.
|
||
This module needs to be visible to both the app and the plugin, in the
|
||
interest of clear and well-defined interfaces:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
module StringProcAPI (Interface(..), plugin) where
|
||
|
||
data Interface = Interface {
|
||
stringProcessor :: String -> String
|
||
}
|
||
|
||
plugin :: Interface
|
||
plugin = Interface { stringProcessor = id }
|
||
}
|
||
\end{quote}
|
||
|
||
Here we define \code{Interface} as the inteface signature for the
|
||
object passed between plugin and application. We'll use the record
|
||
syntax as it looks intuitive in the plugin. We provide a default
|
||
instance, the \code{plugin} value, that can be overwritten in the
|
||
actual plugin, ensuring sensible behaviour in the absence of any
|
||
plugins. The API should theoretically be compiled with \code{-Onot} to
|
||
avoid interface details leaking out into the \code{.hi} file.
|
||
|
||
\subsection*{The Plugin}
|
||
|
||
This is our plugin. Note that the plugin will be compiled entirely
|
||
seperately from the application. It must only rely on the API, and
|
||
nothing in the application source.
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
module StringProcPlugin (resource) where
|
||
|
||
import StringProcAPI (plugin)
|
||
|
||
resource = plugin {
|
||
stringProcessor = reverse
|
||
}
|
||
}
|
||
\end{quote}
|
||
|
||
Using the record syntax we overwrite the \code{function} field with our
|
||
own value, \code{reverse}. The value \code{resource} is the magic symbol
|
||
that must be defined, and which the application will use to find the
|
||
data the plugin exports.
|
||
|
||
Now, we can make this even easier on the plugin writer by the use of a
|
||
``stub'' file. \code{makeWith} lets you merge a plugin source with
|
||
another Haskell file, and compiles the result into the actual plugin
|
||
object. So the application can provide a stub file containing module
|
||
declarations and imports, and a default plugin value. Here is an
|
||
application-provided stub, factoring out compulsory syntax and type
|
||
declarations from the plugin:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
module StringProcPlugin ( resource ) where
|
||
|
||
import StringProcAPI
|
||
|
||
resource :: Interface
|
||
resource = plugin
|
||
}
|
||
\end{quote}
|
||
|
||
By factoring out compulsory syntax, the plugin author only has to
|
||
provide an overriding instance of the \code{resource} field. So all
|
||
the plugin actually consists of, is:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
resource = plugin {
|
||
stringProcessor = reverse
|
||
}
|
||
}
|
||
\end{quote}
|
||
|
||
That is all the code we need! This file may be called anything at all.
|
||
|
||
More complex APIs may have more fields, of course. The nice thing
|
||
about this arrangement is that the user will write some simple syntax,
|
||
which will nonetheless by typechecked safely against the API. Errors
|
||
are also reported using line numbers from the source file, not the
|
||
stub, which makes things less confusing.
|
||
|
||
\subsection*{The Application}
|
||
|
||
Now we need to write an application that can use values of the kind
|
||
defined in the API, and which can compile and load plugins. The basic
|
||
mechanism to compile and load a plugin is as follows:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
do status <- make "StringProcPlugin.hs" []
|
||
obj <- case status of
|
||
MakeSuccess _ o -> return o
|
||
MakeFailure e -> mapM_ putStrLn e >> error "failed"
|
||
|
||
m_v <- load obj ["."] [] "resource"
|
||
val <- case m_v of
|
||
LoadSuccess _ v -> return v
|
||
_ -> error "load failed"
|
||
}
|
||
\end{quote}
|
||
%
|
||
This code calls \code{make} to compile the plugin source, yielding
|
||
wrapper around a handle to an object file. The object can then be loaded
|
||
using \code{load}, and the code associated with the symbol
|
||
\code{resource} is retrieved.
|
||
|
||
We embed this code in a simple shell-like loop, applying the function
|
||
exported by the plugin:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
import Plugins
|
||
import StringProcessorAPI
|
||
import System.Console.Readline
|
||
import System.Exit
|
||
|
||
source = "Plugin.hs"
|
||
stub = "Plugin.stub"
|
||
symbol = "resource"
|
||
|
||
main = do s <- makeWith source stub []
|
||
o <- case s of
|
||
MakeSuccess _ obj -> do
|
||
ls <- load obj ["."] [] symbol
|
||
case ls of LoadSuccess m v -> return (m,v)
|
||
LoadFailure err -> error "load failed"
|
||
MakeFailure e -> mapM_ putStrLn e >> error "compile failed"
|
||
shell o
|
||
|
||
shell o@(m,plugin) = do
|
||
s <- readline "> "
|
||
cmd <- case s of
|
||
Nothing -> exitWith ExitSuccess
|
||
Just (':':'q':_) -> exitWith ExitSuccess
|
||
Just s -> addHistory s >> return s
|
||
|
||
s <- makeWith source stub [] -- maybe recompile the source
|
||
o' <- case s of
|
||
MakeSuccess ReComp o -> do
|
||
ls <- reload m symbol
|
||
case ls of LoadSuccess m' v' -> return (m',v')
|
||
LoadFailure err -> error "reload failed"
|
||
MakeSuccess NotReq _ -> return o
|
||
MakeFailure e -> mapM_ putStrLn e >> shell o
|
||
eval cmd o'
|
||
shell o'
|
||
|
||
eval ":?" _ = putStrLn ":?\n:q\n<string>"
|
||
|
||
eval s (_,plugin) = let fn = (stringProcessor plugin) in putStrLn (fn s)
|
||
}
|
||
\end{quote}
|
||
|
||
We have to import the hs-plugins library, and the API. The main loop
|
||
proceeds by compiling and loading the plugin for the first time, and
|
||
then calls \code{shell}, the interpeter loop. This loop lets us apply
|
||
the function in the plugin to strings we supply. We have to pass
|
||
around the \code{(Module, a)} pair we get back from \code{reload}, so
|
||
that we can pass it to \code{eval} to do the real work. The first
|
||
\code{eval} case is where we use the record syntax to select the
|
||
\code{function} field out of \code{v}, the plugin interface object,
|
||
and we apply it to s. Try it out:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
paprika$ ./a.out
|
||
Loading package base ... linking ... done
|
||
Loading objects API Plugin ... done
|
||
> :?
|
||
":?"
|
||
":q"
|
||
"<string>"
|
||
> abcdefg
|
||
gfedcba
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
Now, if we edit the plugin while the shell is running, the next time
|
||
we type something at the prompt the plugin will be unloaded,
|
||
recompiled and reloaded. Because the plugin is really an EDSL, we can
|
||
use any Haskell we want, so we'll change the plugin to:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
import Data.Char
|
||
|
||
resource = plugin {
|
||
stringProcessor = my_fn
|
||
}
|
||
|
||
my_fn s = map toUpper (reverse s)
|
||
}
|
||
\end{quote}
|
||
|
||
Back to the shell:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
> abcdefg
|
||
Compiling plugin ... done
|
||
Reloading Plugin ... done
|
||
GFEDCBA
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
And that's it: dynamically recompiled and reload Haskell code!
|
||
|
||
\section{Multiple Plugins}
|
||
|
||
It is quite easy to load multiple plugins, that all implement the
|
||
common plugin API, and that all export the same value (though
|
||
implemented differently). This make \hsplugins{} suitable for
|
||
applications that wish to allow an arbitrary number of plugins. The
|
||
main problem with multiple plugins is that they may share
|
||
dependencies, and if \code{load} na\"ively loaded all dependencies
|
||
found in the set of \code{.hi} files associated with all the plugins,
|
||
the GHC rts would crash. To solve this the \hsplugins{} dynamic loader
|
||
maintains state storing a list of what modules and packages have been
|
||
loaded already. If \code{load} is called on a module that is already
|
||
loaded, or dependencies are attempted to load, that have already been
|
||
loaded, the dynamic loader ignores these extra dependencies. This
|
||
makes it quite easy to write an application that will allows an
|
||
arbitrary number of plugins to be loaded. An example follows.
|
||
|
||
\subsection*{Definition}
|
||
|
||
First we need to define the API that a plugin must type check against,
|
||
in order to be valid.
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
module API where
|
||
|
||
data Interface = Interface {
|
||
valueOf :: String -> String
|
||
}
|
||
|
||
plugin :: Interface
|
||
plugin = Interface { valueOf = id }
|
||
}
|
||
\end{quote}
|
||
|
||
We can then implement a number of plugins that provide values of type
|
||
"Interface". We show three plugins that export string manipulation functions:
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
module Plugin1 where
|
||
|
||
import API
|
||
import Data.Char
|
||
|
||
resource = plugin {
|
||
valueOf = \s -> map toUpper s
|
||
}
|
||
}
|
||
\end{quote}
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
module Plugin2 where
|
||
|
||
import API
|
||
import Data.Char
|
||
|
||
resource = plugin {
|
||
valueOf = \s -> map toLower s
|
||
}
|
||
}
|
||
\end{quote}
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
module Plugin3 where
|
||
|
||
import API
|
||
|
||
resource = plugin {
|
||
valueOf = reverse
|
||
}
|
||
}
|
||
\end{quote}
|
||
|
||
And finally we need to write an application that would use these
|
||
plugins. Remember that the application is written without knowledge of
|
||
the plugins, and the plugins are written without knowledge of the
|
||
application. They are each implemented only in terms of the API, a
|
||
shared module and \code{.hi} file. An application needs to make the
|
||
API interface available to plugin authors, by distributing the API
|
||
object file and \code{.hi} file with the application.
|
||
%
|
||
\begin{quote}
|
||
\scm{
|
||
import Plugins
|
||
import API
|
||
|
||
main = do
|
||
let plist = ["Plugin1.o", "Plugin2.o", "Plugin3.o"]
|
||
plugins <- mapM (\p -> load p ["."] [] "resource") plist
|
||
let functions = map (valueOf . fromLoadSuc) plugins
|
||
mapM_ (\f -> putStrLn $ f "haskell is for hackers") functions
|
||
|
||
fromLoadSuc (LoadFailure _) = error "load failed"
|
||
fromLoadSuc (LoadSuccess _ v) = v
|
||
|
||
}
|
||
\end{quote}
|
||
|
||
This application simply loads all the plugins and retrieves the
|
||
functions they export. It then applies each of these functions to a
|
||
string, printing the result. We assume for this example that the
|
||
plugins are compiled once only, and are not compiled dynamically via
|
||
\code{make}. This implies that you have to use \code{GHC} to generate
|
||
the \code{.hi} file for each plugin. A sample Makefile to compile the
|
||
plugins, and the api:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
all:
|
||
ghc -Onot -c API.hs
|
||
ghc -O -c Plugin1.hs
|
||
ghc -O -c Plugin2.hs
|
||
ghc -O -c Plugin3.hs
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
Ghc creates \code{.hi} files for each plugin, which can be inspected
|
||
using the \code{Plugins.BinIface.readBinIface} function. It parses the
|
||
\code{.hi} file, generating, roughly, the following:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
interface "Main" Main
|
||
module dependencies: A, B
|
||
package dependencies: base, haskell98, lang, unix
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
which says that the plugin depends upon a variety of system packages,
|
||
and the modules A and B. All these dependencies must be loaded before
|
||
the plugin itself.
|
||
|
||
You then need to compile the application against the API, and against
|
||
the \hsplugins{} library:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
ghc -O --make -package plugins Main.hs
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
Running the application produces the following result. Note that the
|
||
verbose output can be switched off by compiling \hsplugins{} without
|
||
the \code{-DDEBUG} flag. If you look at the \code{.hi} file, using
|
||
\code{ghc --show-iface}, you'll see that they all depend on the base
|
||
package, and on the API, but the state stored in the dynamic loader
|
||
ensures that these shared modules are only loaded once:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
Loading package base ... linking ... done
|
||
Loading object API Plugin1 ... done
|
||
Loading object Plugin2 ... done
|
||
Loading object Plugin3 ... done
|
||
|
||
HASKELL IS FOR HACKERS
|
||
haskell is for hackers
|
||
srekcah rof si lleksah
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
Archives of plugins can be loaded in one go if they have been linked
|
||
into a .o GHCi package, see \code{loadPackage}.
|
||
|
||
\newpage
|
||
|
||
\appendix
|
||
|
||
\section{License}
|
||
|
||
This library is distributed under the terms of the LGPL:
|
||
|
||
\begin{quote}
|
||
|
||
Copyright 2004, Don Stewart - \url{http://www.cse.unsw.edu.au/~dons}
|
||
|
||
This library is free software; you can redistribute it and/or
|
||
modify it under the terms of the GNU Lesser General Public
|
||
License as published by the Free Software Foundation; either
|
||
version 2.1 of the License, or (at your option) any later version.
|
||
|
||
This library is distributed in the hope that it will be useful,
|
||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||
Lesser General Public License for more details.
|
||
|
||
You should have received a copy of the GNU Lesser General Public
|
||
License along with this library; if not, write to the Free Software
|
||
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
|
||
USA
|
||
|
||
\end{quote}
|
||
|
||
\section{Portability}
|
||
|
||
The library tries to be portable. There are two major points that
|
||
limit easy portabilty. The first is the dependence on the GHC dynamic
|
||
linker. \hsplugins{} is thus limited to platforms to which GHC's dyn
|
||
linker has been ported (this is essentially the same as the platforms
|
||
that can run GHCi).
|
||
|
||
Other than this, there are 3 platform specific items that need to be
|
||
defined for new platforms:
|
||
\begin{itemize}
|
||
\item Where tmp files should be created. Define the
|
||
\code{tmpDir} variable in \code{Plugins/Consts.hs}
|
||
|
||
\item Process creation, such that we can read stdin and stderr
|
||
from the process (this is the POpen library for Posix
|
||
systems). For Windows, \hsplugins{} carries Simon
|
||
Marlow's \code{forkProcess} library.
|
||
|
||
\item Dealing with backslashes in Dos-style path names
|
||
\end{itemize}
|
||
|
||
I plan to solve the above 3 problems (and thus have a Windows port) once
|
||
GHC 6.4 is out.
|
||
|
||
\newpage
|
||
|
||
\section{A Haskell Interpreter using Plugins}
|
||
% \label{sec:plugs}
|
||
|
||
Here is a full length example of a Haskell interpreter/compiler in the
|
||
style of Malcolm Wallace's \code{hi}. Rather than compiling the
|
||
user's code to an executable, we use \hsplugins{} to instead load an
|
||
object file and execute that instead, using the \code{eval} interface.
|
||
This cuts out the linking phase from the process, making turnaround at
|
||
the prompt around twice as fast as \code{hi}.
|
||
|
||
\subsection*{Source of Plugs}
|
||
|
||
\begin{quote}
|
||
\scm{
|
||
import Eval.Haskell
|
||
import Plugins.Load
|
||
|
||
import System.Exit ( ExitCode(..), exitWith )
|
||
import System.IO
|
||
import System.Console.Readline ( readline, addHistory )
|
||
|
||
symbol = "resource"
|
||
|
||
main = do
|
||
putStrLn banner
|
||
putStr "Loading package base" >> hFlush stdout
|
||
loadPackage "base"
|
||
putStr " ... linking ... " >> hFlush stdout
|
||
resolveObjs
|
||
putStrLn "done"
|
||
|
||
shell []
|
||
|
||
shell :: [String] -> IO ()
|
||
shell imps = do
|
||
s <- readline "plugs> "
|
||
cmd <- case s of
|
||
Nothing -> exitWith ExitSuccess
|
||
Just (':':'q':_) -> exitWith ExitSuccess
|
||
Just s -> addHistory s >> return s
|
||
imps' <- run cmd imps
|
||
shell imps'
|
||
|
||
run :: String -> [String] -> IO [String]
|
||
run "" is = return is
|
||
run ":?" is = putStrLn help >> return is
|
||
|
||
run ":l" _ = return []
|
||
run (':':'l':' ':m) is = return (m:is)
|
||
|
||
run (':':'t':' ':s) is = do
|
||
ty <- typeOf s is
|
||
when (not $ null ty) (putStrLn $ s ++ " :: " ++ ty)
|
||
return is
|
||
|
||
run (':':_) is = putStrLn help >> return is
|
||
|
||
run s is = do
|
||
s <- unsafeEval ("show $ "++s) is
|
||
when (isJust s) (putStrLn (fromJust s))
|
||
return is
|
||
|
||
banner = "\
|
||
\ __ \n\
|
||
\ ____ / /_ ______ ______ \n\
|
||
\ / __ \\/ / / / / __ `/ ___/ PLugin User's GHCi System, for Haskell 98\n\
|
||
\ / /_/ / / /_/ / /_/ (__ ) http://www.cse.unsw.edu.au/~dons/hs-plugins\n\
|
||
\ / .___/_/\\__,_/\\__, /____/ Type :? for help \n\
|
||
\/_/ /____/ \n"
|
||
|
||
help = "\
|
||
\Commands :\n\
|
||
\ <expr> evaluate expression\n\
|
||
\ :t <expr> show type of expression (monomorphic only)\n\
|
||
\ :l module bring module in to scope\n\
|
||
\ :l clear module list\n\
|
||
\ :quit quit\n\
|
||
\ :? display this list of commands"
|
||
}
|
||
\end{quote}
|
||
|
||
\subsection*{A Transcript}
|
||
|
||
And a transcript:
|
||
%
|
||
\begin{quote}
|
||
\begin{verbatim}
|
||
$ ./plugs
|
||
__
|
||
____ / /_ ______ ______
|
||
/ __ \/ / / / / __ `/ ___/ PLugin User's GHCi System, for Haskell 98
|
||
/ /_/ / / /_/ / /_/ (__ ) http://www.cse.unsw.edu.au/~dons/hs-plugins
|
||
/ .___/_/\__,_/\__, /____/ Type :? for help
|
||
/_/ /____/
|
||
|
||
Loading package base ... linking ... done
|
||
plugs> 1
|
||
1
|
||
plugs> let x = 1 + 2 in x
|
||
3
|
||
plugs> :l Data.List
|
||
plugs> case [1,3,2] of x -> sort x
|
||
[1,2,3]
|
||
plugs> reverse [1,3,2]
|
||
[2,3,1]
|
||
plugs> map (\x -> (x,2^x)) [1,2,3,4,5,6,7,8,9,10]
|
||
[(1,2),(2,4),(3,8),(4,16),(5,32),(6,64),(7,128),(8,256),(9,512),(10,1024)]
|
||
plugs> :t "haskell"
|
||
"haskell" :: [Char]
|
||
plugs> :quit
|
||
\end{verbatim}
|
||
\end{quote}
|
||
|
||
\end{document}
|