mirror of
https://git.savannah.gnu.org/git/emacs/org-mode.git
synced 2024-09-23 08:10:42 +00:00
adding task "sha1 hash based caching" (thanks to Carsten for the suggestion)
This commit is contained in:
parent
b0d5c57673
commit
d69afdabb6
394
org-babel.org
394
org-babel.org
|
@ -218,51 +218,169 @@ would then be [[#sandbox][the sandbox]].
|
||||||
#+end_src
|
#+end_src
|
||||||
|
|
||||||
|
|
||||||
* Tasks [46/63]
|
* Tasks [47/64]
|
||||||
** PROPOSED allow `anonymous' function block with function call args?
|
** STARTED share org-babel [1/6]
|
||||||
My question here is simply whether we're going to allow
|
how should we share org-babel?
|
||||||
#+begin_src python(arg=ref)
|
*** DONE post to org-mode
|
||||||
# whatever
|
*** TODO post to ess mailing list
|
||||||
|
*** TODO create a org-babel page on worg
|
||||||
|
*** TODO create a short screencast demonstrating org-babel in action
|
||||||
|
*** PROPOSED a peer-reviewed publication?
|
||||||
|
|
||||||
|
The following notes are biased towards statistics-oriented
|
||||||
|
journals because ESS and Sweave are written by people associated
|
||||||
|
with / in statistics departments. But I am sure there are suitable
|
||||||
|
journals out there for an article on using org mode for
|
||||||
|
reproducible research (and literate programming etc).
|
||||||
|
|
||||||
|
Clearly, we would invite Carsten to be involved with this.
|
||||||
|
|
||||||
|
ESS is described in a peer-reviewed journal article:
|
||||||
|
Emacs Speaks Statistics: A Multiplatform, Multipackage Development Environment for Statistical Analysis [Abstract]
|
||||||
|
Journal of Computational & Graphical Statistics 13(1), 247-261
|
||||||
|
Rossini, A.J, Heiberger, R.M., Sparapani, R.A., Maechler, M., Hornik, K. (2004)
|
||||||
|
[[http://www.amstat.org/publications/jcgs.cfm][Journal of Computational and Graphical Statistics]]
|
||||||
|
|
||||||
|
Also [[http://www.amstat.org/publications/jss.cfm][Journal of Statistical Software]] Established in 1996, the
|
||||||
|
Journal of Statistical Software publishes articles, book reviews,
|
||||||
|
code snippets, and software reviews. The contents are freely
|
||||||
|
available online. For both articles and code snippets, the source
|
||||||
|
code is published along with the paper.
|
||||||
|
|
||||||
|
Sweave has a paper:
|
||||||
|
|
||||||
|
Friedrich Leisch and Anthony J. Rossini. Reproducible statistical
|
||||||
|
research. Chance, 16(2):46-50, 2003. [ bib ]
|
||||||
|
|
||||||
|
also
|
||||||
|
|
||||||
|
Friedrich Leisch. Sweave: Dynamic generation of statistical reports
|
||||||
|
using literate data analysis. In Wolfgang Härdle and Bernd Rönz,
|
||||||
|
editors, Compstat 2002 - Proceedings in Computational Statistics,
|
||||||
|
pages 575-580. Physica Verlag, Heidelberg, 2002. ISBN 3-7908-1517-9.
|
||||||
|
|
||||||
|
also
|
||||||
|
|
||||||
|
We could also look at the Journals publishing these [[http://www.reproducibleresearch.net/index.php/RR_links#Articles_about_RR_.28chronologically.29][Reproducible
|
||||||
|
Research articles]].
|
||||||
|
|
||||||
|
*** PROPOSED an article in [[http://journal.r-project.org/][The R Journal]]
|
||||||
|
This looks good. It seems that their main topic to software tools for
|
||||||
|
use by R programmers, and Org-babel is certainly that.
|
||||||
|
|
||||||
|
*** existing similar tools
|
||||||
|
try to collect pointers to similar tools
|
||||||
|
|
||||||
|
Reproducible Research
|
||||||
|
- [[http://en.wikipedia.org/wiki/Sweave][Sweave]]
|
||||||
|
|
||||||
|
Literate Programming
|
||||||
|
- [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]]
|
||||||
|
- [[http://www-cs-faculty.stanford.edu/~knuth/cweb.html][Cweb]]
|
||||||
|
- [[http://www.lri.fr/~filliatr/ocamlweb/][OCamlWeb]]
|
||||||
|
|
||||||
|
Meta Functional Programming
|
||||||
|
- ?
|
||||||
|
|
||||||
|
Programmable Spreadsheet
|
||||||
|
- ?
|
||||||
|
|
||||||
|
*** examples
|
||||||
|
we need to think up some good examples
|
||||||
|
|
||||||
|
**** interactive tutorials
|
||||||
|
This could be a place to use [[* org-babel assertions][org-babel assertions]].
|
||||||
|
|
||||||
|
for example the first step of a tutorial could assert that the version
|
||||||
|
of the software-package (or whatever) is equal to some value, then
|
||||||
|
source-code blocks could be used with confidence (and executed
|
||||||
|
directly from) the rest of the tutorial.
|
||||||
|
|
||||||
|
**** answering a text-book question w/code example
|
||||||
|
org-babel is an ideal environment enabling both the development and
|
||||||
|
demonstrationg of the code snippets required as answers to many
|
||||||
|
text-book questions.
|
||||||
|
|
||||||
|
**** something using tables
|
||||||
|
maybe something along the lines of calculations from collected grades
|
||||||
|
|
||||||
|
**** file sizes
|
||||||
|
Maybe something like the following which outputs sizes of directories
|
||||||
|
under the home directory, and then instead of the trivial =emacs-lisp=
|
||||||
|
block we could use an R block to create a nice pie chart of the
|
||||||
|
results.
|
||||||
|
|
||||||
|
#+srcname: sizes
|
||||||
|
#+begin_src bash :results replace
|
||||||
|
du -sc ~/*
|
||||||
#+end_src
|
#+end_src
|
||||||
|
|
||||||
but with preference given to
|
#+begin_src emacs-lisp :var sizes=sizes :results replace
|
||||||
#+srcname blockname(arg=ref)
|
(mapcar #'car sizes)
|
||||||
** PROPOSED allow :result as synonym for :results?
|
#+end_src
|
||||||
** PROPOSED allow 'output mode to return stdout as value?
|
*** Answer to question on list
|
||||||
Maybe we should allow this. In fact, if block x is called
|
From: Hector Villafuerte <hectorvd@gmail.com>
|
||||||
with :results output, and it references blocks y and z, then
|
Subject: [Orgmode] Merge tables
|
||||||
shouldn't the output of x contain a concatenation of the outputs of
|
Date: Wed, 19 Aug 2009 10:08:40 -0600
|
||||||
y and z, together with x's own output? That would raise the
|
To: emacs-orgmode@gnu.org
|
||||||
question of what happens if y is defined with :results output and z
|
|
||||||
with :results value. I guess z's (possibly vector/tabular) output
|
|
||||||
would be inside a literal example block containing the whole lot.
|
|
||||||
** PROPOSED optional timestamp for output
|
|
||||||
Add option to place an (inactive) timestamp at the #+resname, to
|
|
||||||
record when that output was generated.
|
|
||||||
|
|
||||||
*** source code block timestamps (optional addition)
|
Hi,
|
||||||
[Eric] If we did this would we then want to place a timestamp on the
|
I've just discovered Org and are truly impressed with it; using it for
|
||||||
source-code block, so that we would know if the results are
|
more and more tasks.
|
||||||
current or out of date? This would have the effect of caching the
|
|
||||||
results of calculations and then only re-running if the
|
|
||||||
source-code has changed. For the caching to work we would need to
|
|
||||||
check not only the timestamp on a source-code block, but also the
|
|
||||||
timestamps of any tables or source-code blocks referenced by the
|
|
||||||
original source-code block.
|
|
||||||
|
|
||||||
[Dan] I do remember getting frustrated by Sweave always having to
|
Here's what I want to do: I have 2 tables with the same number of rows
|
||||||
re-do everything, so this could be desirable, as long as it's easy
|
(one row per subject). I would like to make just one big table by
|
||||||
to over-ride of course. I'm not sure it should be the default
|
copying the second table to the right of the first one. This is a
|
||||||
behaviour unless we are very confident that it works well.
|
no-brainer in a spreadsheet but my attempts in Org have failed. Any
|
||||||
|
ideas?
|
||||||
|
|
||||||
**** maintaining source-code block timestamps
|
By the way, thanks for this great piece of software!
|
||||||
It may make sense to add a hook to `org-edit-special' which could
|
--
|
||||||
update the source-code blocks timestamp. If the user edits the
|
hector
|
||||||
contents of a source-code block directly I can think of no
|
|
||||||
efficient way of maintaining the timestamp.
|
**** Suppose the tables are as follows
|
||||||
** TODO make tangle files read-only?
|
#+tblname: tab1
|
||||||
With a file-local variable setting, yea that makes sense. Maybe
|
| a | b | c |
|
||||||
the header should reference the related org-mode file.
|
|---+---+---|
|
||||||
|
| 1 | 2 | 3 |
|
||||||
|
| 7 | 8 | 9 |
|
||||||
|
|
||||||
|
#+tblname: tab2
|
||||||
|
| d | e | f |
|
||||||
|
|----+----+----|
|
||||||
|
| 4 | 5 | 6 |
|
||||||
|
| 10 | 11 | 12 |
|
||||||
|
|
||||||
|
**** Here is an answer using R in org-babel
|
||||||
|
|
||||||
|
#+srcname: column-bind(a=tab1, b=tab2)
|
||||||
|
#+begin_src R :colnames t
|
||||||
|
cbind(a, b)
|
||||||
|
#+end_src
|
||||||
|
|
||||||
|
#+resname: column-bind
|
||||||
|
| "a" | "b" | "c" | "d" | "e" | "f" |
|
||||||
|
|-----+-----+-----+-----+-----+-----|
|
||||||
|
| 1 | 2 | 3 | 4 | 5 | 6 |
|
||||||
|
| 7 | 8 | 9 | 10 | 11 | 12 |
|
||||||
|
|
||||||
|
|
||||||
|
**** Alternatively
|
||||||
|
Use org-table-export, do it in external spreadsheet software,
|
||||||
|
then org-table-import
|
||||||
|
** TODO sha1 hash based caching
|
||||||
|
:PROPERTIES:
|
||||||
|
:CUSTOM_ID: sha1-caching
|
||||||
|
:END:
|
||||||
|
|
||||||
|
#+begin_quote
|
||||||
|
I wonder if we should consider some cashing of images, also for
|
||||||
|
export. I think we could have an alist with sha1 hashes as keys and
|
||||||
|
image files as values. The sha1 hash could be made from the entire
|
||||||
|
code and the command that is used to create the image..
|
||||||
|
|
||||||
|
-- Carsten
|
||||||
|
#+end_quote
|
||||||
|
|
||||||
** TODO support for working with =*Org Edit Src Example*= buffers [4/6]
|
** TODO support for working with =*Org Edit Src Example*= buffers [4/6]
|
||||||
*** STARTED Patch against org source.
|
*** STARTED Patch against org source.
|
||||||
|
@ -604,155 +722,6 @@ msg + " y python"
|
||||||
#+begin_src emacs-lisp
|
#+begin_src emacs-lisp
|
||||||
(concat msg " elisp")
|
(concat msg " elisp")
|
||||||
#+end_src
|
#+end_src
|
||||||
** STARTED share org-babel [1/6]
|
|
||||||
how should we share org-babel?
|
|
||||||
*** DONE post to org-mode
|
|
||||||
*** TODO post to ess mailing list
|
|
||||||
*** TODO create a org-babel page on worg
|
|
||||||
*** TODO create a short screencast demonstrating org-babel in action
|
|
||||||
*** PROPOSED a peer-reviewed publication?
|
|
||||||
|
|
||||||
The following notes are biased towards statistics-oriented
|
|
||||||
journals because ESS and Sweave are written by people associated
|
|
||||||
with / in statistics departments. But I am sure there are suitable
|
|
||||||
journals out there for an article on using org mode for
|
|
||||||
reproducible research (and literate programming etc).
|
|
||||||
|
|
||||||
Clearly, we would invite Carsten to be involved with this.
|
|
||||||
|
|
||||||
ESS is described in a peer-reviewed journal article:
|
|
||||||
Emacs Speaks Statistics: A Multiplatform, Multipackage Development Environment for Statistical Analysis [Abstract]
|
|
||||||
Journal of Computational & Graphical Statistics 13(1), 247-261
|
|
||||||
Rossini, A.J, Heiberger, R.M., Sparapani, R.A., Maechler, M., Hornik, K. (2004)
|
|
||||||
[[http://www.amstat.org/publications/jcgs.cfm][Journal of Computational and Graphical Statistics]]
|
|
||||||
|
|
||||||
Also [[http://www.amstat.org/publications/jss.cfm][Journal of Statistical Software]] Established in 1996, the
|
|
||||||
Journal of Statistical Software publishes articles, book reviews,
|
|
||||||
code snippets, and software reviews. The contents are freely
|
|
||||||
available online. For both articles and code snippets, the source
|
|
||||||
code is published along with the paper.
|
|
||||||
|
|
||||||
Sweave has a paper:
|
|
||||||
|
|
||||||
Friedrich Leisch and Anthony J. Rossini. Reproducible statistical
|
|
||||||
research. Chance, 16(2):46-50, 2003. [ bib ]
|
|
||||||
|
|
||||||
also
|
|
||||||
|
|
||||||
Friedrich Leisch. Sweave: Dynamic generation of statistical reports
|
|
||||||
using literate data analysis. In Wolfgang Härdle and Bernd Rönz,
|
|
||||||
editors, Compstat 2002 - Proceedings in Computational Statistics,
|
|
||||||
pages 575-580. Physica Verlag, Heidelberg, 2002. ISBN 3-7908-1517-9.
|
|
||||||
|
|
||||||
also
|
|
||||||
|
|
||||||
We could also look at the Journals publishing these [[http://www.reproducibleresearch.net/index.php/RR_links#Articles_about_RR_.28chronologically.29][Reproducible
|
|
||||||
Research articles]].
|
|
||||||
|
|
||||||
*** PROPOSED an article in [[http://journal.r-project.org/][The R Journal]]
|
|
||||||
This looks good. It seems that their main topic to software tools for
|
|
||||||
use by R programmers, and Org-babel is certainly that.
|
|
||||||
|
|
||||||
*** existing similar tools
|
|
||||||
try to collect pointers to similar tools
|
|
||||||
|
|
||||||
Reproducible Research
|
|
||||||
- [[http://en.wikipedia.org/wiki/Sweave][Sweave]]
|
|
||||||
|
|
||||||
Literate Programming
|
|
||||||
- [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]]
|
|
||||||
- [[http://www-cs-faculty.stanford.edu/~knuth/cweb.html][Cweb]]
|
|
||||||
- [[http://www.lri.fr/~filliatr/ocamlweb/][OCamlWeb]]
|
|
||||||
|
|
||||||
Meta Functional Programming
|
|
||||||
- ?
|
|
||||||
|
|
||||||
Programmable Spreadsheet
|
|
||||||
- ?
|
|
||||||
|
|
||||||
*** examples
|
|
||||||
we need to think up some good examples
|
|
||||||
|
|
||||||
**** interactive tutorials
|
|
||||||
This could be a place to use [[* org-babel assertions][org-babel assertions]].
|
|
||||||
|
|
||||||
for example the first step of a tutorial could assert that the version
|
|
||||||
of the software-package (or whatever) is equal to some value, then
|
|
||||||
source-code blocks could be used with confidence (and executed
|
|
||||||
directly from) the rest of the tutorial.
|
|
||||||
|
|
||||||
**** answering a text-book question w/code example
|
|
||||||
org-babel is an ideal environment enabling both the development and
|
|
||||||
demonstrationg of the code snippets required as answers to many
|
|
||||||
text-book questions.
|
|
||||||
|
|
||||||
**** something using tables
|
|
||||||
maybe something along the lines of calculations from collected grades
|
|
||||||
|
|
||||||
**** file sizes
|
|
||||||
Maybe something like the following which outputs sizes of directories
|
|
||||||
under the home directory, and then instead of the trivial =emacs-lisp=
|
|
||||||
block we could use an R block to create a nice pie chart of the
|
|
||||||
results.
|
|
||||||
|
|
||||||
#+srcname: sizes
|
|
||||||
#+begin_src bash :results replace
|
|
||||||
du -sc ~/*
|
|
||||||
#+end_src
|
|
||||||
|
|
||||||
#+begin_src emacs-lisp :var sizes=sizes :results replace
|
|
||||||
(mapcar #'car sizes)
|
|
||||||
#+end_src
|
|
||||||
*** Answer to question on list
|
|
||||||
From: Hector Villafuerte <hectorvd@gmail.com>
|
|
||||||
Subject: [Orgmode] Merge tables
|
|
||||||
Date: Wed, 19 Aug 2009 10:08:40 -0600
|
|
||||||
To: emacs-orgmode@gnu.org
|
|
||||||
|
|
||||||
Hi,
|
|
||||||
I've just discovered Org and are truly impressed with it; using it for
|
|
||||||
more and more tasks.
|
|
||||||
|
|
||||||
Here's what I want to do: I have 2 tables with the same number of rows
|
|
||||||
(one row per subject). I would like to make just one big table by
|
|
||||||
copying the second table to the right of the first one. This is a
|
|
||||||
no-brainer in a spreadsheet but my attempts in Org have failed. Any
|
|
||||||
ideas?
|
|
||||||
|
|
||||||
By the way, thanks for this great piece of software!
|
|
||||||
--
|
|
||||||
hector
|
|
||||||
|
|
||||||
**** Suppose the tables are as follows
|
|
||||||
#+tblname: tab1
|
|
||||||
| a | b | c |
|
|
||||||
|---+---+---|
|
|
||||||
| 1 | 2 | 3 |
|
|
||||||
| 7 | 8 | 9 |
|
|
||||||
|
|
||||||
#+tblname: tab2
|
|
||||||
| d | e | f |
|
|
||||||
|----+----+----|
|
|
||||||
| 4 | 5 | 6 |
|
|
||||||
| 10 | 11 | 12 |
|
|
||||||
|
|
||||||
**** Here is an answer using R in org-babel
|
|
||||||
|
|
||||||
#+srcname: column-bind(a=tab1, b=tab2)
|
|
||||||
#+begin_src R :colnames t
|
|
||||||
cbind(a, b)
|
|
||||||
#+end_src
|
|
||||||
|
|
||||||
#+resname: column-bind
|
|
||||||
| "a" | "b" | "c" | "d" | "e" | "f" |
|
|
||||||
|-----+-----+-----+-----+-----+-----|
|
|
||||||
| 1 | 2 | 3 | 4 | 5 | 6 |
|
|
||||||
| 7 | 8 | 9 | 10 | 11 | 12 |
|
|
||||||
|
|
||||||
|
|
||||||
**** Alternatively
|
|
||||||
Use org-table-export, do it in external spreadsheet software,
|
|
||||||
then org-table-import
|
|
||||||
** TODO command line execution
|
** TODO command line execution
|
||||||
Allow source code blocks to be called form the command line. This
|
Allow source code blocks to be called form the command line. This
|
||||||
will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
|
will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
|
||||||
|
@ -788,6 +757,26 @@ should use a span class, and should show original source in tool-tip
|
||||||
** TODO LoB: re-implement plotting and analysis functions from org-R
|
** TODO LoB: re-implement plotting and analysis functions from org-R
|
||||||
I'll do this soon, now that we things are a bit more settled and we
|
I'll do this soon, now that we things are a bit more settled and we
|
||||||
have column names in R.
|
have column names in R.
|
||||||
|
** PROPOSED allow `anonymous' function block with function call args?
|
||||||
|
My question here is simply whether we're going to allow
|
||||||
|
#+begin_src python(arg=ref)
|
||||||
|
# whatever
|
||||||
|
#+end_src
|
||||||
|
|
||||||
|
but with preference given to
|
||||||
|
#+srcname blockname(arg=ref)
|
||||||
|
** PROPOSED allow :result as synonym for :results?
|
||||||
|
** PROPOSED allow 'output mode to return stdout as value?
|
||||||
|
Maybe we should allow this. In fact, if block x is called
|
||||||
|
with :results output, and it references blocks y and z, then
|
||||||
|
shouldn't the output of x contain a concatenation of the outputs of
|
||||||
|
y and z, together with x's own output? That would raise the
|
||||||
|
question of what happens if y is defined with :results output and z
|
||||||
|
with :results value. I guess z's (possibly vector/tabular) output
|
||||||
|
would be inside a literal example block containing the whole lot.
|
||||||
|
** PROPOSED make tangle files read-only?
|
||||||
|
With a file-local variable setting, yea that makes sense. Maybe
|
||||||
|
the header should reference the related org-mode file.
|
||||||
** PROPOSED Creating presentations
|
** PROPOSED Creating presentations
|
||||||
The [[mairix:t:@@9854.1246500519@gamaville.dokosmarshall.org][recent thread]] containing posts by Nick Dokos and Sebastian
|
The [[mairix:t:@@9854.1246500519@gamaville.dokosmarshall.org][recent thread]] containing posts by Nick Dokos and Sebastian
|
||||||
Vaubán on exporting to beamer looked very interesting, but I
|
Vaubán on exporting to beamer looked very interesting, but I
|
||||||
|
@ -839,6 +828,33 @@ the org-mode buffer as a link to the file...
|
||||||
This would allow for display of images upon export providing
|
This would allow for display of images upon export providing
|
||||||
functionality similar to =org-exp-blocks= only in a more general
|
functionality similar to =org-exp-blocks= only in a more general
|
||||||
manner.
|
manner.
|
||||||
|
** DEFERRED optional timestamp for output
|
||||||
|
*DEFERRED*: I'm deferring this in deference to the better caching
|
||||||
|
system proposed by Carsten. (see [[sha1-caching]])
|
||||||
|
|
||||||
|
Add option to place an (inactive) timestamp at the #+resname, to
|
||||||
|
record when that output was generated.
|
||||||
|
|
||||||
|
*** source code block timestamps (optional addition)
|
||||||
|
[Eric] If we did this would we then want to place a timestamp on the
|
||||||
|
source-code block, so that we would know if the results are
|
||||||
|
current or out of date? This would have the effect of caching the
|
||||||
|
results of calculations and then only re-running if the
|
||||||
|
source-code has changed. For the caching to work we would need to
|
||||||
|
check not only the timestamp on a source-code block, but also the
|
||||||
|
timestamps of any tables or source-code blocks referenced by the
|
||||||
|
original source-code block.
|
||||||
|
|
||||||
|
[Dan] I do remember getting frustrated by Sweave always having to
|
||||||
|
re-do everything, so this could be desirable, as long as it's easy
|
||||||
|
to over-ride of course. I'm not sure it should be the default
|
||||||
|
behaviour unless we are very confident that it works well.
|
||||||
|
|
||||||
|
**** maintaining source-code block timestamps
|
||||||
|
It may make sense to add a hook to `org-edit-special' which could
|
||||||
|
update the source-code blocks timestamp. If the user edits the
|
||||||
|
contents of a source-code block directly I can think of no
|
||||||
|
efficient way of maintaining the timestamp.
|
||||||
** DEFERRED figure out how to handle errors during evaluation
|
** DEFERRED figure out how to handle errors during evaluation
|
||||||
I expect it will be hard to do this properly, but ultimately it
|
I expect it will be hard to do this properly, but ultimately it
|
||||||
would be nice to be able to specify somewhere to receive STDERR,
|
would be nice to be able to specify somewhere to receive STDERR,
|
||||||
|
|
Loading…
Reference in a new issue