trimming down the size of contrib/babel
This commit is contained in:
Eric Schulte
parent
eec9c235d0
commit
2d2a2a1956
File diff suppressed because it is too large
Load Diff
|
|
@ -1,726 +0,0 @@
|
|||
#+TITLE: Org-Babel Requirements
|
||||
#+OPTIONS: toc:nil num:nil ^:nil
|
||||
|
||||
This file contains the initial discussion of the requirements for
|
||||
org-babel.
|
||||
|
||||
* Overview
|
||||
This project is basically about putting source code into org
|
||||
files. This isn't just code to look pretty as a source code example,
|
||||
but code to be evaluated. Org files have 3 main export targets: org,
|
||||
html and latex. Once we have implemented a smooth bi-directional flow
|
||||
of data between org-mode formats (including tables, and maybe lists
|
||||
and property values) and source-code blocks, we will be able to use
|
||||
org-mode's built in export to publish the results of evaluated source
|
||||
code in any org-supported format using org-mode as an intermediate
|
||||
format. We have a current focus on R code, but we are regarding that
|
||||
more as a working example than as a defining feature of the project.
|
||||
|
||||
The main objectives of this project are...
|
||||
|
||||
# Lets start with this list and make changes as appropriate. Please
|
||||
# try to make changes to this list, rather than starting any new
|
||||
# lists.
|
||||
|
||||
- [[* evaluation of embedded source code][evaluation of embedded source code]]
|
||||
- [[* execution on demand and on export][execution on demand and on export]]
|
||||
- [[* source blocks][source blocks]]
|
||||
- [[* header arguments][header arguments]]
|
||||
- [[* inline source evaluation][inline source evaluation]]
|
||||
- [[* included source file evaluation][included source file evaluation]] ?? maybe
|
||||
- [[* caching of evaluation][caching of evaluation]]
|
||||
- [[* interaction with the source-code's process][interaction with the source-code's process]]
|
||||
- [[* output of code evaluation][output of code evaluation]]
|
||||
- [[* textual/numeric output][textual/numeric output]]
|
||||
- [[* graphical output][graphical output]]
|
||||
- [[* file creation][non-graphics file creation]]
|
||||
- [[* side effects][side effects]]
|
||||
- [[* reference to data and evaluation results][reference to data and evaluation results]]
|
||||
- [[* reference format][reference format]]
|
||||
- [[* source-target pairs][source-target pairs]]
|
||||
- [[* source block output from org tables][source block output from org tables]]
|
||||
- [[* source block outpt from other source block][source block outpt from other source block]]
|
||||
- [[* source block output from org list][source block output from org list]] ?? maybe
|
||||
- [[* org table from source block][org table from source block]]
|
||||
- [[* org table from org table][org table from org table]]
|
||||
- [[* org properties from source block][org properties from source block]]
|
||||
- [[* org properties from org table][org properties from org table]]
|
||||
- [[* export][export]]
|
||||
|
||||
|
||||
* Objectives and Specs
|
||||
|
||||
** evaluation of embedded source code
|
||||
|
||||
*** execution on demand and on export
|
||||
Let's use an asterisk to indicate content which includes the
|
||||
*result* of code evaluation, rather than the code itself. Clearly
|
||||
we have a requirement for the following transformation:
|
||||
|
||||
org \to org*
|
||||
|
||||
Let's say this transformation is effected by a function
|
||||
`org-eval-buffer'. This transformation is necessary when the
|
||||
target format is org (say you want to update the values in an org
|
||||
table, or generate a plot and create an org link to it), and it
|
||||
can also be used as the first step by which to reach html and
|
||||
latex:
|
||||
|
||||
org \to org* \to html
|
||||
|
||||
org \to org* \to latex
|
||||
|
||||
Thus in principle we can reach our 3 target formats with
|
||||
`org-eval-buffer', `org-export-as-latex' and `org-export-as-html'.
|
||||
|
||||
An extra transformation that we might want is
|
||||
|
||||
org \to latex
|
||||
|
||||
I.e. export to latex without evaluation of code, in such a way that R
|
||||
code can subsequently be evaluated using
|
||||
=Sweave(driver=RweaveLatex)=, which is what the R community is
|
||||
used to. This would provide a `bail out' avenue where users can
|
||||
escape org mode and enter a workflow in which the latex/noweb file
|
||||
is treated as source.
|
||||
|
||||
**** How do we implement `org-eval-buffer'?
|
||||
|
||||
AIUI The following can all be viewed as implementations of
|
||||
org-eval-buffer for R code:
|
||||
|
||||
(see this question again posed in [[file:org-babel/org-babel-R.el::Maybe%20the%20following%20be%20replaced%20with%20a%20method%20using%20ess%20execute][org-babel-R.el]])
|
||||
|
||||
***** org-eval-light
|
||||
This is the beginnings of a general evaluation mechanism, that
|
||||
could evaluate python, ruby, shell, perl, in addition to R.
|
||||
The header says it's based on org-eval
|
||||
|
||||
what is org-eval??
|
||||
|
||||
org-eval was written by Carsten. It lives in the
|
||||
org/contrib/lisp directory because it is too dangerous to
|
||||
include in the base. Unlike org-eval-light org-eval evaluates
|
||||
all source blocks in an org-file when the file is first opened,
|
||||
which could be a security nightmare for example if someone
|
||||
emailed you a pernicious file.
|
||||
|
||||
***** org-R
|
||||
This accomplishes org \to org* in elisp by visiting code blocks
|
||||
and evaluating code using ESS.
|
||||
|
||||
***** RweaveOrg
|
||||
This accomplishes org \to org* using R via
|
||||
|
||||
: Sweave("file-with-unevaluated-code.org", driver=RweaveOrg, syntax=SweaveSyntaxOrg)
|
||||
|
||||
***** org-exp-blocks.el
|
||||
Like org-R, this achieves org \to org* in elisp by visiting code
|
||||
blocks and using ESS to evaluate R code.
|
||||
|
||||
*** source blocks
|
||||
(see [[* Special editing and evaluation of source code][Special editing and evaluation of source code]])
|
||||
|
||||
*** header arguments
|
||||
(see [[* block headers/parameters][block headers/parameters]])
|
||||
|
||||
There are going to be many cases where we want to use header arguments
|
||||
to change the evaluation options of source code, to pass external
|
||||
information to a block of source code and control the inclusion of
|
||||
evaluation results.
|
||||
|
||||
*** inline source evaluation
|
||||
*** included source file evaluation
|
||||
It may be nice to be able to include an entire external file of source
|
||||
code, and then evaluate and export that code as if it were in the
|
||||
file. The format for such a file inclusion could optionally look like
|
||||
the following
|
||||
|
||||
: #+include_src filename header_arguments
|
||||
|
||||
*** caching of evaluation
|
||||
|
||||
Any kind of code that can have a block evaluated could optionally define
|
||||
a function to write the output to a file, or to serialize the output of
|
||||
the function. If a document or block is configured to cache input,
|
||||
write all cached blocks to their own files and either a) hash them, or
|
||||
b) let git and org-attach track them. Before a block gets eval'd, we
|
||||
check to see if it has changed. If a document or block is configured to
|
||||
cache output and a print/serialize function is available, write the
|
||||
output of each cached block to its own file. When the file is eval'd
|
||||
and some sort of display is called for, only update the display if the
|
||||
output has changed. Each of these would have an override, presumably
|
||||
something like (... & force) that could be triggered with a prefix arg
|
||||
to the eval or export function.
|
||||
|
||||
For R, I would say
|
||||
|
||||
#+begin_src emacs-lisp
|
||||
;; fake code that only pretends to work
|
||||
(add-hook 'rorg-store-output-hook
|
||||
'("r" lambda (block-environment block-label)
|
||||
(ess-exec (concat "save.image("
|
||||
block-environment
|
||||
", file = " block-label
|
||||
".Rdata, compress=TRUE)"))))
|
||||
#+end_src
|
||||
|
||||
The idea being that for r blocks that get eval'd, if output needs to be
|
||||
stored, you should write the entire environment that was created in that
|
||||
block to an Rdata file.
|
||||
|
||||
(see [[* block scoping][block scoping]])
|
||||
|
||||
** interaction with the source-code's process
|
||||
We should settle on a uniform API for sending code and receiving
|
||||
output from a source process. Then to add a new language all we need
|
||||
to do is implement this API.
|
||||
|
||||
for related notes see ([[* Interaction with the R process][Interaction with the R process]])
|
||||
|
||||
** output of code evaluation
|
||||
*** textual/numeric output
|
||||
We (optionally) incorporate the text output as text in the target
|
||||
document
|
||||
*** graphical output
|
||||
We either link to the graphics or (html/latex) include them
|
||||
inline.
|
||||
|
||||
I would say, if the block is being evaluated interactively then
|
||||
lets pop up the image in a new window, and if it is being exported
|
||||
then we can just include a link to the file which will be exported
|
||||
appropriately by org-mode.
|
||||
|
||||
*** non-graphics files
|
||||
? We link to other file output
|
||||
*** side effects
|
||||
If we are using a continuous process in (for example an R process
|
||||
handled by ESS) then any side effects of the process (for example
|
||||
setting values of R variables) will be handled automatically
|
||||
|
||||
Are there side-effects which need to be considered aside from those
|
||||
internal to the source-code evaluation process?
|
||||
|
||||
** reference to data and evaluation results
|
||||
I think this will be very important. I would suggest that since we
|
||||
are using lisp we use lists as our medium of exchange. Then all we
|
||||
need are functions going converting all of our target formats to and
|
||||
from lists. These functions are already provided by for org tables.
|
||||
|
||||
It would be a boon both to org users and R users to allow org tables
|
||||
to be manipulated with the R programming language. Org tables give R
|
||||
users an easy way to enter and display data; R gives org users a
|
||||
powerful way to perform vector operations, statistical tests, and
|
||||
visualization on their tables.
|
||||
|
||||
This means that we will need to consider unique id's for source
|
||||
blocks, as well as for org tables, and for any other data source or
|
||||
target.
|
||||
|
||||
*** Implementations
|
||||
**** naive
|
||||
Naive implementation would be to use =(org-export-table "tmp.csv")=
|
||||
and =(ess-execute "read.csv('tmp.csv')")=.
|
||||
**** org-R
|
||||
org-R passes data to R from two sources: org tables, or csv
|
||||
files. Org tables are first exported to a temporary csv file
|
||||
using [[file:existing_tools/org-R.el::defun%20org%20R%20export%20to%20csv%20csv%20file%20options][org-R-export-to-csv]].
|
||||
**** org-exp-blocks
|
||||
org-exp-blocks uses [[org-interblock-R-command-to-string]] to send
|
||||
commands to an R process running in a comint buffer through ESS.
|
||||
org-exp-blocks has no support for dumping table data to R process, or
|
||||
vice versa.
|
||||
|
||||
**** RweaveOrg
|
||||
NA
|
||||
|
||||
*** reference format
|
||||
This will be tricky, Dan has already come up with a solution for R, I
|
||||
need to look more closely at that and we should try to come up with a
|
||||
formats for referencing data from source-code in such a way that it
|
||||
will be as source-code-language independent as possible.
|
||||
|
||||
Org tables already have a sophisticated reference system in place
|
||||
that allows referencing table ranges in other files, as well as
|
||||
specifying constants in the header arguments of a table. This is
|
||||
described in [[info:org:References]].
|
||||
|
||||
**** Dan: thinking aloud re: referencing data from R
|
||||
Suppose in some R code, we want to reference data in an org
|
||||
table. I think that requires the use of 'header arguments', since
|
||||
otherwise, under pure evaluation of a code block without header
|
||||
args, R has no way to locate the data in the org buffer. So that
|
||||
suggests a mechanism like that used by org-R whereby table names
|
||||
or unique entry IDs are used to reference org tables (and indeed
|
||||
potentially row/column ranges within org tables, although that
|
||||
subsetting could also be done in R).
|
||||
|
||||
Specifically what org-R does is write the table to a temp csv
|
||||
file, and tell R the name of that file. However:
|
||||
|
||||
1. We are not limited to a single source of input; the same sort
|
||||
of thing could be done for several sources of input
|
||||
|
||||
2. I don't think we even have to use temp files. An alternative
|
||||
would be to have org pass the table contents as a csv-format
|
||||
string to textConnection() in R, thus creating an arbitrary
|
||||
number of input objects in the appropriate R environment
|
||||
(scope) from which the R code can read data when necessary.
|
||||
|
||||
That suggests a header option syntax something like
|
||||
|
||||
#+begin_src emacs-lisp
|
||||
'(:R-obj-name-1 tbl-name-or-id-1 :R-obj-name-2 tbl-name-or-id-2)
|
||||
#+end_src
|
||||
|
||||
As a result of passing that option, the code would be able to access
|
||||
the data referenced by table-name-or-id-2 via read.table(R-obj-name-1).
|
||||
|
||||
An extension of that idea would be to allow remote files to be used as
|
||||
data sources. In this case one might need just the remote file (if
|
||||
it's a csv file), or if it's an org file then the name of the file
|
||||
plus a table reference within that org file. Thus maybe something like
|
||||
|
||||
#+begin_src emacs-lisp
|
||||
'((R-obj-name-1 . (:tblref tbl-name-or-id-1 :file file-1))
|
||||
(R-obj-name-2 . (:tblref tbl-name-or-id-2 :file file-2)))
|
||||
#+end_src emacs-lisp
|
||||
|
||||
**** Eric: referencing data in general
|
||||
So here's some thoughts for referencing data (henceforth referred to
|
||||
as *resources*). I think this is the next thing we need to tackle for
|
||||
implementation to move forward. We don't need to implement everything
|
||||
below right off the bat, but I'd like to get these lists as full as
|
||||
possible so we don't make any implementation assumptions which
|
||||
preclude real needs.
|
||||
|
||||
We need to reference resources of the following types...
|
||||
|
||||
- table (list)
|
||||
- output from a source code block (list or hash)
|
||||
- property values of an outline header (hash)
|
||||
- list (list)
|
||||
- description list (hash)
|
||||
- more?...
|
||||
|
||||
All of these resources will live in org files which could be
|
||||
|
||||
- the current file (default)
|
||||
- another file on the same system (path)
|
||||
- another file on the web (url)
|
||||
- another file in a git repo (file and commit hash)
|
||||
|
||||
What information should each of these resources be able to supply?
|
||||
I'm thinking (again not that we'll implement all of these but just to
|
||||
think of them)...
|
||||
|
||||
- ranges or points of vector data
|
||||
- key/value pairs from a hash
|
||||
- when the object was last modified
|
||||
- commit info (author, date, message, sha, etc...)
|
||||
- pointers to the resources upon which the resource relies
|
||||
|
||||
So we need a referencing syntax powerful enough to handle all of these
|
||||
alternatives. Maybe something like =path:sha:name:range= where
|
||||
|
||||
- path :: is empty for the current file, is a path for files on the
|
||||
same system, and is a url otherwise
|
||||
- sha :: is an option git commit indicator
|
||||
- name :: is the table/header/source-block name or id for location
|
||||
inside of the org file (this would not be optional)
|
||||
- range :: would indicate which information is requested from the
|
||||
resource, so it could be a range to access parts of a
|
||||
table, or the names of properties to be referenced from an
|
||||
outline header
|
||||
|
||||
Once we agree on how this should work, I'll try to stub out some code,
|
||||
so that we can get some simple subset of this functionality working,
|
||||
hopefully something complex enough to do the following...
|
||||
- [[* resource reference example][resource-reference-example]]
|
||||
|
||||
***** questions
|
||||
****** multiple outputs
|
||||
Do we want things like a source code block to leave multiple outputs,
|
||||
or do we only want them to be able to output one vector or hash?
|
||||
|
||||
****** environment (state and side-effects)
|
||||
This design assumes that any changes will explicitly pass data in a
|
||||
functional programming style. This makes no assumptions about things
|
||||
like source code blocks changing state (in general state changes lead
|
||||
to more difficult debugging).
|
||||
|
||||
- Do we want to take steps so ensure we do things like execute
|
||||
consecutive R blocks in different environment, or do we want to
|
||||
allow state changes?
|
||||
- Does this matter?
|
||||
|
||||
****** passing arguments to resources
|
||||
So I(eric) may be getting ahead of myself here, but what do you think
|
||||
about the ability to pass arguments to resources. I'm having visions
|
||||
of google map-reduce, processes spread out across multiple machines.
|
||||
|
||||
Maybe we could do this by allowing the arguments to be specified?
|
||||
|
||||
*** source-target pairs
|
||||
|
||||
The following can be used for special considerations based on
|
||||
source-target pairs
|
||||
|
||||
Dan: I don't quite understand this subtree; Eric -- could you give
|
||||
a little more explanation of this and of your comment above
|
||||
regarding using [[lists as our medium of exchange]]?
|
||||
|
||||
**** source block output from org tables
|
||||
**** source block outpt from other source block
|
||||
**** source block output from org list
|
||||
**** org table from source block
|
||||
**** org table from org table
|
||||
**** org properties from source block
|
||||
**** org properties from org table
|
||||
|
||||
** export
|
||||
once the previous objectives are met export should be fairly simple.
|
||||
Basically it will consist of triggering the evaluation of source code
|
||||
blocks with the org-export-preprocess-hook.
|
||||
|
||||
This block export evaluation will be aware of the target format
|
||||
through the htmlp and latexp variables, and can then create quoted
|
||||
=#+begin_html= and =#+begin_latex= blocks appropriately.
|
||||
|
||||
There will also need to be a set of header arguments related to
|
||||
code export. These would be similar to the results header
|
||||
arguments but would apply to how to handle execution and results
|
||||
during export.
|
||||
|
||||
|
||||
* Notes
|
||||
** Block Formats
|
||||
Unfortunately org-mode how two different block types, both useful.
|
||||
In developing RweaveOrg, a third was introduced.
|
||||
|
||||
Eric is leaning towards using the =#+begin_src= blocks, as that is
|
||||
really what these blocks contain: source code. Austin believes
|
||||
that specifying export options at the beginning of a block is
|
||||
useful functionality, to be preserved if possible.
|
||||
|
||||
Note that upper and lower case are not relevant in block headings.
|
||||
|
||||
*** PROPOSED block format
|
||||
I (Eric) propose that we use the syntax of source code blocks as they
|
||||
currently exist in org-mode with the addition of *evaluation*,
|
||||
*header-arguments*, *exportation*, *single-line-blocks*, and
|
||||
*references-to-table-data*.
|
||||
|
||||
1) *evaluation*: These blocks can be evaluated through =\C-c\C-c= with
|
||||
a slight addition to the code already present and working in
|
||||
[[file:existing_tools/org-eval-light.el][org-eval-light.el]]. All we should need to add for R support would
|
||||
be an appropriate entry in [[org-eval-light-interpreters]] with a
|
||||
corresponding evaluation function. For an example usinga
|
||||
org-eval-light see [[* src block evaluation w/org-eval-light]].
|
||||
|
||||
2) *header-arguments*: These can be implemented along the lines of
|
||||
Austin's header arguments in [[file:existing_tools/RweaveOrg/org-sweave.el][org-sweave.el]].
|
||||
|
||||
3) *exportation*: Should be as similar as possible to that done by
|
||||
Sweave, and hopefully can re-use some of the code currently present
|
||||
in [[file:existing_tools/exp-blocks/org-exp-blocks.el ][org-exp-blocks.el]].
|
||||
|
||||
4) *single-line-blocks*: It seems that it is useful to be able to
|
||||
place a single line of R code on a line by itself. Should we add
|
||||
syntax for this similar to Dan's =#+RR:= lines? I would lean
|
||||
towards something here that can be re-used for any type of source
|
||||
code in the same manner as the =#+begin_src R= blocks, maybe
|
||||
=#+src_R=? Dan: I'm fine with this, but don't think single-line
|
||||
blocks are a priority. My =#+R= lines were something totally
|
||||
different: an attempt to have users specify R code implicitly,
|
||||
using org-mode option syntax.
|
||||
|
||||
5) *references-to-table-data*: I get this impression that this is
|
||||
vital to the efficient use of R code in an org file, so we should
|
||||
come up with a way to reference table data from a single-line-block
|
||||
or from an R source-code block. It looks like Dan has already done
|
||||
this in [[file:existing_tools/org-R.el][org-R.el]].
|
||||
|
||||
Syntax
|
||||
|
||||
Multi-line Block
|
||||
: #+begin_src lang header-arguments
|
||||
: body
|
||||
: #+end
|
||||
- lang :: the language of the block (R, shell, elisp, etc...)
|
||||
- header-arguments :: a list of optional arguments which control how
|
||||
the block is evaluated and exported, and how the results are handled
|
||||
- body :: the actual body of the block
|
||||
|
||||
Single-line Block
|
||||
: #+begin_src lang body
|
||||
- It's not clear how/if we would include header-arguments into a
|
||||
single line block. Suggestions? Can we just leave them out? Dan:
|
||||
I'm not too worried about single line blocks to start off
|
||||
with. Their main advantage seems to be that they save 2 lines.
|
||||
Eric: Fair enough, lets not worry about this now, also I would guess
|
||||
that any code simple enough to fit on one line wouldn't need header
|
||||
arguments anyways.
|
||||
|
||||
Include Block
|
||||
: #+include_src lang filename header-arguments
|
||||
- I think this would be useful, and should be much more work (Dan:
|
||||
didn't get the meaning of that last clause!?). Eric: scratch that,
|
||||
I meant "*shouldn't* be too much work" :) That way whole external
|
||||
files of source code could be evaluated as if they were an inline
|
||||
block. Dan: again I'd say not a massive priority, as I think all the
|
||||
languages we have in mind have facilities for doing this natively,
|
||||
thus I think the desired effect can often be achieved from within a
|
||||
#+begin_src block. Eric: Agreed, while this would be a nice thing
|
||||
to include we shouldn't wast too much effort on it in the beginning.
|
||||
|
||||
What do you think? Does this accomplish everything we want to be able
|
||||
to do with embedded R source code blocks?
|
||||
|
||||
***** src block evaluation w/org-eval-light
|
||||
here's an example using org-eval-light.el
|
||||
|
||||
first load the org-eval-light.el file
|
||||
|
||||
[[elisp:(load (expand-file-name "org-eval-light.el" (expand-file-name "existing_tools" (file-name-directory buffer-file-name))))]]
|
||||
|
||||
then press =\C-c\C-c= inside of the following src code snippet. The
|
||||
results should appear in a comment immediately following the source
|
||||
code block. It shouldn't be too hard to add R support to this
|
||||
function through the `org-eval-light-interpreters' variable.
|
||||
|
||||
(Dan: The following causes error on export to HTML hence spaces inserted at bol)
|
||||
|
||||
#+begin_src shell
|
||||
date
|
||||
#+end_src
|
||||
|
||||
*** existing formats
|
||||
**** Source code blocks
|
||||
Org has an extremely useful method of editing source code and
|
||||
examples in their native modes. In the case of R code, we want to
|
||||
be able to use the full functionality of ESS mode, including
|
||||
interactive evaluation of code.
|
||||
|
||||
Source code blocks look like the following and allow for the
|
||||
special editing of code inside of the block through
|
||||
`org-edit-special'.
|
||||
|
||||
#+BEGIN_SRC r
|
||||
|
||||
,## hit C-c ' within this block to enter a temporary buffer in r-mode.
|
||||
|
||||
,## while in the temporary buffer, hit C-c C-c on this comment to
|
||||
,## evaluate this block
|
||||
a <- 3
|
||||
a
|
||||
|
||||
,## hit C-c ' to exit the temporary buffer
|
||||
#+END_SRC
|
||||
|
||||
**** dblocks
|
||||
dblocks are useful because org-mode will automatically call
|
||||
`org-dblock-write:dblock-type' where dblock-type is the string
|
||||
following the =#+BEGIN:= portion of the line.
|
||||
|
||||
dblocks look like the following and allow for evaluation of the
|
||||
code inside of the block by calling =\C-c\C-c= on the header of
|
||||
the block.
|
||||
|
||||
#+BEGIN: dblock-type
|
||||
#+END:
|
||||
|
||||
**** R blocks
|
||||
In developing RweaveOrg, Austin created [[file:existing_tools/RweaveOrg/org-sweave.el][org-sweave.el]]. This
|
||||
allows for the kind of blocks shown in [[file:existing_tools/RweaveOrg/testing.Rorg][testing.Rorg]]. These blocks
|
||||
have the advantage of accepting options to the Sweave preprocessor
|
||||
following the #+BEGIN_R declaration.
|
||||
|
||||
*** block headers/parameters
|
||||
Regardless of the syntax/format chosen for the source blocks, we will
|
||||
need to be able to pass a list of parameters to these blocks. These
|
||||
should include (but should certainly not be limited to)
|
||||
- label or id :: Label of the block, should we provide facilities for
|
||||
automatically generating a unique one of these?
|
||||
- file :: names of file to which graphical/textual/numerical/tabular output
|
||||
should be written. Do we need this, or should this be controlled
|
||||
through the source code itself?
|
||||
- results :: indication of where the results should be placed, maybe
|
||||
the following values...
|
||||
- append :: *default* meaning just append to the current buffer
|
||||
immediately following the current source block
|
||||
- replace :: like append, but replace any results currently there
|
||||
- file :: save the results in a new file, and place a link to the
|
||||
file into the current buffer immediately following the
|
||||
source code block
|
||||
- table :: save the results into a table, maybe use a table id:range
|
||||
to identify which table and where therein
|
||||
- nil :: meaning just discard the results
|
||||
- not sure of a good name here :: flags for when/if the block should
|
||||
be evaluated (on export etc...)
|
||||
- again can't thing of a concise name :: flags for how the results of
|
||||
the export should be displayed/included
|
||||
- scope :: flag indicating whether the block should have a local or
|
||||
global scope
|
||||
- flags specific to the language of the source block
|
||||
- etc...
|
||||
|
||||
I think fleshing out this list is an important next step.
|
||||
|
||||
** Interaction with the R process
|
||||
|
||||
We should take care to implement this in such a way that all of the
|
||||
different components which have to interactive with R including:
|
||||
- evaluation of source code blocks
|
||||
- automatic evaluation on export
|
||||
- evaluation of \R{} snippets
|
||||
- evaluation of single source code lines
|
||||
- evaluation of included source code files
|
||||
- sending/receiving vector data
|
||||
|
||||
I think we currently have two implementations of interaction with R
|
||||
processes; [[file:existing_tools/org-R.el][org-R.el]] and [[file:existing_tools/exp-blocks/org-exp-blocks.el ][org-exp-blocks.el]]. We should be sure to take
|
||||
the best of each of these approaches.
|
||||
|
||||
More on the exchange of data at between org-mode and source code
|
||||
blocks at [[* reference to data and evaluation results][reference to data and evaluation results]].
|
||||
|
||||
** block scoping
|
||||
(see [[* caching of evaluation][caching of evaluation]])
|
||||
|
||||
This inadvertently raises the issue of scoping. The pretend function
|
||||
pretends that we will create a block-local scope, and that we can save
|
||||
just the things in that scope. Sweave takes the make-everything-global
|
||||
approach. I can see advantages either way. If we make block-local
|
||||
scopes, we can save each one independently, and generally speaking it
|
||||
seems like more granularity==more control. If we make everything
|
||||
global, we can refer to entities declared in earlier blocks without
|
||||
having to explicitly import those entities into the current block. I
|
||||
think this counts in the "need to think about it early on" category.
|
||||
|
||||
If we did want block-local scopes, in R we can start every eval with
|
||||
something like
|
||||
|
||||
;; fake code that pretends to create a new, empty environment
|
||||
(ess-exec (concat block-env " <- new.env()"))
|
||||
(ess-exec (concat "eval(" block-contents ", envir=" block-env ")"))
|
||||
|
||||
If we decide we want block-scoping, I'm sure Dan and I can figure out
|
||||
the right way to do this in R, if he hasn't already. I haven't thought
|
||||
at all about how these scope issues generalize to, say, bash blocks.
|
||||
|
||||
Maybe this is something that should be controlled by a header
|
||||
argument?
|
||||
|
||||
** =\C-c\C-c= evaluation
|
||||
|
||||
With org-mode version at least 6.23, see the documentation for
|
||||
[[info:org:Context-sensitive%20commands][info:org:Context-sensitive commands]].
|
||||
|
||||
** free explicit variables
|
||||
Maybe we should have some idea of variables independent of any
|
||||
particular type of source code or source block. These could be
|
||||
variables that have a value inside of the scope of the org-mode file,
|
||||
and they could be used as a transport mechanism for information
|
||||
transfer between org-tables, org-lists, and different source-blocks.
|
||||
|
||||
Each type of source code (and org-mode types like tables, lists,
|
||||
etc...) would need to implement functions for converting different
|
||||
types of data to and from these variables (which would be elisp
|
||||
variables).
|
||||
|
||||
So for example say we want to read the values from a table into an R
|
||||
block, perform some calculations, and then write the results back into
|
||||
the table. We could
|
||||
1) assign the table to a variable
|
||||
- the table would be converted into a lisp vector (list of lists)
|
||||
- the vector would be saved in the variable
|
||||
2) an R source block would reference the variable
|
||||
- the variable would be instantiated into an R variable (through
|
||||
mechanisms mentioned [[* Dan: thinking aloud re: referencing data from R][elsewhere]])
|
||||
- the R code is executed, and the value of the variable *inside of
|
||||
R* is updated
|
||||
- when the R block finished the value of the variable *globally in
|
||||
the org buffer* would be updated
|
||||
3) optionally the global value of the variable would be converted back
|
||||
into an org-mode table and would be used to overwrite the existing
|
||||
table.
|
||||
|
||||
What do you think?
|
||||
|
||||
This might not be too different from what we were already talking
|
||||
about, but I think the introduction of the idea of having variables
|
||||
existing independently of any tables or source code blocks is novel
|
||||
and probably has some advantages (and probably shortfalls).
|
||||
|
||||
|
||||
* COMMENT Commentary
|
||||
I'm seeing this as like commit notes, and a place for less formal
|
||||
communication of the goals of our changes.
|
||||
|
||||
** Eric <2009-02-06 Fri 15:41>
|
||||
I think we're getting close to a comprehensive set of objectives
|
||||
(although since you two are the real R user's I leave that decision up
|
||||
to you). Once we've agreed on a set of objectives and agreed on at
|
||||
least to broad strokes of implementation, I think we should start
|
||||
listing out and assigning tasks.
|
||||
|
||||
** Eric <2009-02-09 Mon 14:25>
|
||||
I've done a fairly destructive edit of this file. The main goal was
|
||||
to enforce a structure on the document that we can use moving forward,
|
||||
so that any future objective changes are all made to the main
|
||||
objective list.
|
||||
|
||||
I apologize for removing sections written by other people. I did this
|
||||
when they were redundant or it was not clear how to fit them into this
|
||||
structure. Rest assured if the previous text wasn't persisted in git
|
||||
I would have been much more cautious about removing it.
|
||||
|
||||
I hope that this outline structure should be able to remain stable
|
||||
through the process of fleshing out objectives, and cashing those
|
||||
objectives out into tasks. That said, please feel free to make any
|
||||
changes that you see fit.
|
||||
|
||||
** Dan <2009-02-12 Thu 10:23>
|
||||
Good job Eric with major works on this file.
|
||||
|
||||
** Eric <2009-02-22 Sun 13:17>
|
||||
So I skipped ahead and got started on the fun part. Namely stubbing
|
||||
out some of the basic functionality. Please don't take any of the
|
||||
decisions I've made so far (on things like names, functionality,
|
||||
design etc...) as final decisions, I'm of course open to and hoping
|
||||
for improvement.
|
||||
|
||||
So far [[file:org-babel/org-babel.el][org-babel.el]] and [[file:org-babel/org-babel-script.el][org-babel-script.el]] can be used to evaluate source
|
||||
code blocks of simple scripting languages. It shouldn't be too hard
|
||||
(any takers) to write a org-babel-R.el modeled after org-babel-script.el
|
||||
to use for evaluating R code files.
|
||||
|
||||
See the [[* org-babel.el beginning functionality][Sandbox]] for evaluable examples.
|
||||
|
||||
** Eric <2009-02-23 Mon 15:12>
|
||||
While thinking about how to implement the transfer of data between
|
||||
source blocks and the containing org-mode file, I decided it *might*
|
||||
be useful to explicitly support the existence of variables which exist
|
||||
independent of source blocks or tables. I'd appreciate any
|
||||
feedback... (see [[free explicit variables][free explicit variables]])
|
||||
|
||||
** Eric <2009-02-23 Mon 17:53>
|
||||
So as I start populating this file with source code blocks I figure I
|
||||
should share this... I don't know if you guys use [[http://code.google.com/p/smart-snippet/][yasnippet]] at all,
|
||||
but if you do you might find this [[file:block][block-snippet]] org-mode snippet
|
||||
useful (I use it all the time).
|
||||
|
||||
** Dan <2009-05-14 Thu 19:13>
|
||||
Please note that I am at an early stage of learning org-babel /
|
||||
studying the code so I may be misunderstanding things.
|
||||
1. removed prefix arg [[2. evaluate the transpose definition =\C-c\C-c= on the beginning of][here]]
|
||||
2. changed :replace -> replace
|
||||
3. added some tasks and bugs
|
||||
|
||||
|
||||
* Buffer Dictionary
|
||||
LocalWords: DBlocks dblocks org-babel el eric fontification
|
||||
|
||||
|
||||
|
||||
Loading…
Reference in New Issue