#+OPTIONS: H:3 num:nil toc:2 \n:nil @:t ::t |:t ^:{} -:t f:t *:t TeX:t LaTeX:t skip:nil d:(HIDE) tags:not-in-toc #+STARTUP: align fold nodlcheck hidestars oddeven lognotestate hideblocks #+SEQ_TODO: TODO(t) INPROGRESS(i) WAITING(w@) | DONE(d) CANCELED(c@) #+TAGS: Write(w) Update(u) Fix(f) Check(c) #+TITLE: Org-babel #+AUTHOR: Eric Schulte, Dan Davison #+EMAIL: schulte.eric at gmail dot com, davison at stats dot ox dot ac dot uk #+LANGUAGE: en #+CATEGORY: worg # #+INFOJS_OPT: view:content #+begin_html
executable source code blocks in org-mode
#+end_html * Introduction :PROPERTIES: :CUSTOM_ID: introduction :END: Org-babel is an extension to the very excellent [[http://orgmode.org/][Org-mode]]; an [[http://www.gnu.org/software/emacs/][Emacs]] major mode for doing almost anything with plain text. If you are not familiar with Org-mode please take a moment to read [[http://orgmode.org/][the Org-mode homepage]] before continuing. Org-babel provides the following modifications to [[http://orgmode.org/manual/Literal-examples.html][the existing support]] for blocks of source code examples in the org-mode core. 1. Interactive source code execution 2. Arguments to source code blocks 3. Exportation of source code blocks to files (literate programming) * Getting started :PROPERTIES: :CUSTOM_ID: getting-started :END: 1) Grab the latest code from the git repo at [[http://github.com/eschulte/org-babel/tree/master][github/org-babel]] #+begin_src sh git clone git://github.com/eschulte/org-babel.git #+end_src 2) Add the following lines to your .emacs, replacing the path as appropriate. A good place to check that things are up and running would the examples in [[* Basic org-babel functionality][Basic org-babel functionality]]. #+begin_src emacs-lisp (add-to-list 'load-path "/path/to/org-babel/lisp") (require 'org-babel-init) #+end_src 3) Finally, activate the subset of supported Org-babel languages which you want to be able to execute on your system. As an example, the following activates python, ruby and R. For a full list of languages, with notes on their dependencies see the [[#reference-and-documentation][Reference / Documentation]] section below. #+begin_src emacs-lisp (require 'org-babel-python) (require 'org-babel-ruby) (require 'org-babel-R) ;; ;; Once you've activated languages, load the library of babel to ;; make pre-built helper functions available in the languages you will be using. (org-babel-load-library-of-babel) #+end_src * Basic org-babel functionality :PROPERTIES: :CUSTOM_ID: basic-functionality :END: *** Source code blocks :PROPERTIES: :CUSTOM_ID: source-code-blocks :END: Org-babel is all about *source code blocks* in org mode. These are blocks of code (in whatever language), that can occur anywhere in an org-mode file. For example, the following is a source block containing [[http://www.ruby-lang.org/][ruby]] code: : #+begin_src ruby : "This file was last evaluated on #{Date.today}" : #+end_src If you are unfamiliar with the notion of a source code block in org-mode, please have a look at the [[http://orgmode.org/manual/Literal-examples.html][relevant manual section]] before proceding. Note that above is what the source block looks like in the org-mode file. We had to take [[http://orgmode.org/manual/Literal-examples.html#Literal-examples][special steps]] to make it look that way in the HTML output. Normally, when exported to HTML, source blocks are fontified according to their language, and the begin_src...end_src mark-up is omitted, like this: #+begin_src ruby "This file was last evaluated on #{Date.today}" #+end_src From now on, if you are viewing the HTML version, you will see the HTML output only. However, much of this document consists of interactive examples, and therefore in order to get a feeling for the mechanics of Org-babel it might make most sense to grab the plain text version of this file #+HTML: org-babel-worg.org and work through it in Emacs. Alternatively the htmlized version of the plain text of this file at #+HTML: org-babel-worg.html allows the plain text version to be viewed (non-interactively) in a web browser. *** Source code execution :PROPERTIES: :CUSTOM_ID: source-code-execution :END: For interpreted languages such as shell, python, R, etc, org-babel allows source blocks to be executed: the code is passed to the interpreter and you have control over what is done with the results of excecution. Here are three examples of code blocks in three different languages, followed by their output. If you are viewing the plain text version of this document in emacs, place point anywhere inside the blocks and use =C-c C-c= to run the code[fn:1] (and feel free to alter it!). **** Ruby #+begin_src ruby "This file was last evaluated on #{Date.today}" #+end_src #+resname: : This file was last evaluated on 2009-08-09 **** [[http://www.r-project.org/][R]] #+begin_src R :results value matrix(rnorm(6), nrow=2) #+end_src #+resname: | -0.138279734486552 | -2.2476234005706 | -0.0839549402407832 | | 0.0730510956002737 | 0.0634015508602321 | 0.174013159381603 | **** [[http://ditaa.sourceforge.net/][ditaa]] #+begin_src ditaa :file images/blue.png :cmdline -r +---------+ | cBLU | | | | +----+ | |cPNK| | | | +----+----+ #+end_src #+resname: [[file:images/blue.png]] *** Source code block syntax The basic syntax of source-code blocks in Org-babel is as follows: : #+srcname: name(arguments) : #+begin_src language header-arguments : body : #+end_src - name :: This name is associated with the source-code block. This is similar to the =#+tblname= lines which can be used to name tables in org-mode files. By referencing the srcname of a source-code block it is possible to evaluate the block from other places, files, or from inside tables. - arguments :: Code blocks can have arguments (see [[#arguments-to-source-code-blocks][below]]) which are provided using a familiar function-call syntax similar to (e.g.) python or R. - language :: The language of the code in the source-code block. Valid values must be members of `org-babel-interpreters'. - header-arguments :: Header arguments control many facets of the evaluation and output of source-code blocks. See the [[* Header Arguments][Header Arguments]] section for a complete review of available header arguments. - body :: The actual source code which will be evaluated. An important key-binding to become familiar with is =C-c '=. This calls `org-edit-special' which brings up an edit buffer containing the code using the emacs major mode appropriate to the language. *** What happens to the results? :PROPERTIES: :CUSTOM_ID: results :END: Org-babel provides two fundamentally different modes for capturing the results of code evaluation, specified by the =:results= header argument. **** =:results value= (functional mode) This means that the 'result' of code evaluation is defined to be the *value* of the last statement in the block. Thus with this setting, one can view the code block as a function with a return value. And not only can you view it that way, but you can actually use the return value of one source block as input for another (see later). This setting is the default. As an example, consider the following block of python code and its output. #+begin_src python :results value import time print("Hello, today's date is %s" % time.ctime()) print('Two plus two is') 2 + 2 #+end_src #+resname: : 4 Notice that in functional mode, the output consists of the value of the last statement, and nothing else. **** =:results output= (scripting mode) With this setting, org-babel captures all the text output of the code block and places it in the org buffer. One can think of this as a 'scripting' mode: the code block contains a series of commands, and you get the output of all the commands. Unlike in the 'functional' mode, the code block has no return value. (This mode will be more familiar to Sweave users). Now consider the result of evaluating the same source block as before, but under scripting mode. #+srcname: name #+begin_src python :results output import time print("Hello, today's date is %s" % time.ctime()) print('Two plus two is') 2 + 2 #+end_src #+resname: name : Hello, today's date is Fri Sep 4 19:49:06 2009 : Two plus two is Again, we got what we asked for: all the text output (stdout) from python. Since we didn't print the last value (2 + 2), we didn't get it in our output. *** Arguments to source code blocks :PROPERTIES: :CUSTOM_ID: arguments-to-source-code-blocks :END: In addition to evaluation of code blocks, org-babel allows them to be parameterised (i.e. have arguments). Thus source code blocks now have the status of *functions*. Arguments to code blocks can be used in both functional and scripting mode. **** Simple example of using a source block as a function First let's look at a very simple example. The following source block defines an org-babel function that will square its input. #+srcname: square(x) #+begin_src python x*x #+end_src In the org-mode file that looks like this: : #+srcname: square(x) : #+begin_src python : x*x : #+end_src Now we use the source block: : #+lob: square(x=6) #+lob: square(x=6) #+resname: square(x=6) : 36 **** A more complex example: using an org-table as input In this example we're going to define a function to compute a Fibonacci sequence, and we're going to make it take its input from a table in the org-mode buffer. Here are the inputs for fibonacci-seq: #+tblname: fibonacci-inputs | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 | in the Org-mode buffer this looks like : #+tblname: fibonacci-inputs : | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | : | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 | [[http://www.gnu.org/software/emacs/manual/elisp.html][Emacs Lisp]] source code #+srcname: fibonacci-seq(fib-inputs=fibonacci-inputs) #+begin_src emacs-lisp (defun fibonacci (n) (if (or (= n 0) (= n 1)) n (+ (fibonacci (- n 1)) (fibonacci (- n 2))))) (mapcar (lambda (row) (mapcar #'fibonacci row)) fib-inputs) #+end_src in the Org-mode buffer this looks like : #+srcname: fibonacci-seq(fib-inputs=fibonacci-inputs) : #+begin_src emacs-lisp : (defun fibonacci (n) : (if (or (= n 0) (= n 1)) : n : (+ (fibonacci (- n 1)) (fibonacci (- n 2))))) : : (mapcar (lambda (row) : (mapcar #'fibonacci row)) fib-inputs) : #+end_src Results of Emacs Lisp code evaluation #+resname: | 1 | 1 | 2 | 3 | 5 | 8 | 13 | 21 | 34 | 55 | | 1 | 3 | 8 | 21 | 55 | 144 | 377 | 987 | 2584 | 6765 | * A meta-programming language for org-mode :PROPERTIES: :CUSTOM_ID: meta-programming-language :END: Since information can pass freely between source-code blocks and org-mode tables you can mix and match languages using each language for those tasks to which it is suited. This makes Org-mode files with Org-babel into a kind of meta-functional programming language in which functions from many languages can work together. As an example, lets take some system diagnostics in the shell, and then graph them with R. 1. Shell source code #+srcname: directories #+begin_src bash :results replace cd ~ && du -sc * |grep -v total #+end_src 2. Results of the shell source code (on my system, grab this org-mode files and try running it on your own) #+resname: directories | 72 | "Desktop" | | 12156104 | "Documents" | | 3482440 | "Downloads" | | 2901720 | "Library" | | 57344 | "Movies" | | 16548024 | "Music" | | 120 | "News" | | 7649472 | "Pictures" | | 0 | "Public" | | 152224 | "Sites" | | 8 | "System" | | 56 | "bin" | | 3821872 | "mail" | | 10605392 | "src" | | 1264 | "tools" | 3. R source code (which calls the previous shell source code) #+srcname: directory-pie #+begin_src R :var dirs = directories :session R-pie-example pie(dirs[,1], labels = dirs[,2]) #+end_src 4. Results of R code [[file:images/dirs.png]] * Spreadsheet plugins for org-mode in any language :PROPERTIES: :CUSTOM_ID: spreadsheet :END: *NOTE*: Maybe in-addition-to/in-stead-of this example we should do a more traditional "spreadsheet" example with R [Eric] Not only can Org-babel pass entire tables of data to source code blocks (see [[arguments-to-source-code-blocks]]), Org-babel can also be used to call source code blocks from *within* tables using the Org-mode's [[http://orgmode.org/manual/The-spreadsheet.html#The-spreadsheet][existing spreadsheet functionality]]. In fact the functional test suite for Org-babel is implemented as a large Org-mode table. To run the entire test suite you simple evaluate the table =C-u C-c C-c=, and all of the tests are run updating the table with pass/fail statistics. Here's a sample of our test suite. #+TBLNAME: org-babel-tests | functionality | block | arg | expected | results | pass | |------------------+--------------+-----+-------------+-------------+------| | basic evaluation | | | | | pass | |------------------+--------------+-----+-------------+-------------+------| | emacs lisp | basic-elisp | 2 | 4 | 4 | pass | | shell | basic-shell | | 6 | 6 | pass | | ruby | basic-ruby | | org-babel | org-babel | pass | | python | basic-python | | hello world | hello world | pass | | R | basic-R | | 13 | 13 | pass | #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5)) #+TBLFM: $5=""::$6="" *** code blocks for tests #+srcname: basic-elisp #+begin_src emacs-lisp :var n=7 (* 2 n) #+end_src #+srcname: basic-shell #+begin_src sh :results silent expr 1 + 5 #+end_src #+srcname: date-simple #+begin_src sh :results silent date #+end_src #+srcname: basic-ruby #+begin_src ruby :results silent "org-babel" #+end_src #+srcname: basic-python #+begin_src python :results silent 'hello world' #+end_src #+srcname: basic-R #+begin_src R :results silent b <- 9 b + 4 #+end_src * Library of Babel :PROPERTIES: :CUSTOM_ID: library-of-babel :END: What about those source code blocks which are so useful you want to have them available in every org-mode buffer? The [[file:library-of-babel.org][Library of Babel]] is an extensible collection of ready-made and easily-shortcut-callable source-code blocks for handling common tasks. Org-babel comes pre-populated with the source-code blocks located in the [[file:library-of-babel.org][library-of-babel.org]] file. It is possible to add source-code blocks from any org-mode file to the library by calling #+srcname: add-file-to-lob #+begin_src emacs-lisp (org-babel-lob-ingest "path/to/file.org") #+end_src * Reproducible Research :PROPERTIES: :CUSTOM_ID: reproducable-research :END: #+begin_quote An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures. -- D. Donoho #+end_quote [[http://reproducibleresearch.net/index.php/Main_Page][Reproducible Research]] (RR) is the practice of distributing along with an article of research all data, code, and tools required to reproduce the results discussed in the paper. As such the paper becomes not only a document describing the research but a complete laboratory in which the research can be reproduced and extended. Org-mode already has exceptional support for [[http://orgmode.org/manual/Exporting.html#Exporting][exporting to html and LaTeX]]. Org-babel makes Org-mode a tool for RR by *activating* the data and source code embedded into Org-mode documents making the entire document executable. This makes it not only possible, but natural to distribute research in a format that encourages readers to recreate your results, and perform their own analysis. One notable existing RR tool is [[http://en.wikipedia.org/wiki/Sweave][Sweave]] which provides for the embedding of [[http://www.r-project.org/][R]] code into LaTeX documents. While Sweave is a mature and very useful tool, we believe that Org-babel has several advantages: - It supports multiple languages (we're not aware of other RR tools that do this) - The [[http://orgmode.org/manual/Exporting.html#Exporting][export process]] is flexible and powerful, including HTML as a target in addition to LaTeX - The document can make native use of all the features of Org-mode, such as those for [[http://orgmode.org/manual/Agenda-Views.html#Agenda-Views][project planning]] and [[http://orgmode.org/manual/TODO-Items.html#TODO-Items][task management]] * Literate programming :PROPERTIES: :CUSTOM_ID: literate-programming :END: #+begin_quote Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a /computer/ what to do, let us concentrate rather on explaining to /human beings/ what we want a computer to do. The practitioner of literate programming can be regarded as an essayist, whose main concern is with exposition and excellence of style. Such an author, with thesaurus in hand, chooses the names of variables carefully and explains what each variable means. He or she strives for a program that is comprehensible because its concepts have been introduced in an order that is best for human understanding, using a mixture of formal and informal methods that reinforce each other. -- Donald Knuth #+end_quote Org-babel supports [[http://en.wikipedia.org/wiki/Literate_programming][Literate Programming]] (LP) by allowing the act of programming to take place inside of Org-mode documents. The Org-mode file can then be exported (*woven* in LP speak) to html or LaTeX for consumption by a human, and the embedded source code can be extracted (*tangled* in LP speak) into structured source code files for consumption by a computer. To support these operations Org-babel relies on Org-mode's [[http://orgmode.org/manual/Exporting.html#Exporting][existing exporting functionality]] for *weaving* of documentation, and on the =org-babel-tangle= function which makes use of [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]] [[reference-expansion][reference syntax]] for *tangling* of code files. The [[literate-programming-example][following example]] demonstrates the process of *tangling* in Org-babel. *** Simple Literate Programming Example (Noweb syntax) :PROPERTIES: :CUSTOM_ID: literate-programming-example :END: Tangling functionality is controlled by the =tangle= family of [[header-arguments]]. These arguments can be used to turn tangling on or off (the default) on the source code block, or the outline heading level. The following demonstrates the combination of three source code blocks into a single source code file using =org-babel-tangle=. The following two blocks will not be tangled by default since they have no =tangle= header arguments. #+srcname: hello-world-prefix #+begin_src sh :exports none echo "/-----------------------------------------------------------\\" #+end_src : #+srcname: hello-world-prefix : #+begin_src sh :exports none : echo "/-----------------------------------------------------------\\" : #+end_src #+srcname: hello-world-postfix #+begin_src sh :exports none echo "\-----------------------------------------------------------/" #+end_src : #+srcname: hello-world-postfix : #+begin_src sh :exports none : echo "\-----------------------------------------------------------/" : #+end_src The third block does have a =tangle= header argument indicating the name of the file to which it should be written. It also has [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]] style references to the two previous source code blocks which will be expanded during tangling to include them in the output file as well. #+srcname: hello-world #+begin_src sh :tangle hello :exports none # <