From e4d14d41ecab3c21412181886921248b19135552 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Sun, 19 Jul 2009 13:55:42 -0400 Subject: [PATCH 1/8] This should have been part of the parent commit (911e77a7c966ec5d073c67bc153e197d5edb9fd9) --- lisp/langs/org-babel-R.el | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lisp/langs/org-babel-R.el b/lisp/langs/org-babel-R.el index b401a4073..c2ca67936 100644 --- a/lisp/langs/org-babel-R.el +++ b/lisp/langs/org-babel-R.el @@ -92,7 +92,7 @@ called by `org-babel-execute-src-block' via multiple-value-bind." (defvar org-babel-R-wrapper-method "main <- function ()\n{\n%s\n} write.table(main(), file=\"%s\", sep=\"\\t\", na=\"nil\",row.names=FALSE, col.names=%s, quote=FALSE)") -(defun org-babel-R-evaluate (buffer body result-type colnames) +(defun org-babel-R-evaluate (buffer body result-type column-names-p) "Pass BODY to the R process in BUFFER. If RESULT-TYPE equals 'output then return a list of the outputs of the statements in BODY, if RESULT-TYPE equals 'value then return the value of the From aa1747b0656edf50691dd7ebae7c6ab9789f25f9 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Sun, 19 Jul 2009 13:56:38 -0400 Subject: [PATCH 2/8] Quite a few revisions / additions to the tasks and bugs lists. --- lisp/langs/org-babel-gnuplot.el | 6 +- org-babel.org | 475 ++++++++++++++++++++++---------- 2 files changed, 331 insertions(+), 150 deletions(-) diff --git a/lisp/langs/org-babel-gnuplot.el b/lisp/langs/org-babel-gnuplot.el index 3f587ed8b..766c27a7c 100644 --- a/lisp/langs/org-babel-gnuplot.el +++ b/lisp/langs/org-babel-gnuplot.el @@ -48,9 +48,11 @@ (defun org-babel-execute:gnuplot (body params) "Execute a block of Gnuplot code with org-babel. This function is -called by `org-babel-execute-src-block'." +called by `org-babel-execute-src-block' via multiple-value-bind." (message "executing Gnuplot source code block") - (let* ((vars (org-babel-ref-variables params)) + (let* (;; should be able to lose the next two lines but I don't know + ;; how to test it so not doing it now. + (vars (org-babel-ref-variables params)) (result-params (split-string (or (cdr (assoc :results params)) ""))) (out-file (cdr (assoc :file params))) (cmdline (cdr (assoc :cmdline params))) diff --git a/org-babel.org b/org-babel.org index 7af3f8c00..9d78445d9 100644 --- a/org-babel.org +++ b/org-babel.org @@ -207,7 +207,24 @@ would then be [[#sandbox][the sandbox]]. #+end_src -* Tasks [32/50] +* Tasks [35/55] +** PROPOSED allow `anonymous' function block with function call args? + My question here is simply whether we're going to allow +#+begin_src python(arg=ref) +# whatever +#+end_src + +but with preference given to +#+srcname blockname(arg=ref) +** PROPOSED allow :result as synonym for :results? +** PROPOSED allow 'output mode to return stdout as value? + Maybe we should allow this. In fact, if block x is called + with :results output, and it references blocks y and z, then + shouldn't the output of x contain a concatenation of the outputs of + y and z, together with x's own output? That would raise the + question of what happens if y is defined with :results output and z + with :results value. I guess z's (possibly vector/tabular) output + would be inside a literal example block containing the whole lot. ** PROPOSED optional timestamp for output Add option to place an (inactive) timestamp at the #+resname, to record when that output was generated. @@ -357,6 +374,13 @@ org-mode core similar status to a source code block? - Would be nice to allow org and non-org files to be remote ** TODO figure out how to handle errors during evaluation + I expect it will be hard to do this properly, but ultimately it + would be nice to be able to specify somewhere to receive STDERR, + and to be warned if it is non-empty. + + Probably simpler in non-session evaluation than session? At least + the mechanism will be different I guess. + R has a try function, with error handling, along the lines of python. I bet ruby does too. Maybe more of an issue for functional style; in my proposed scripting style the error just gets dumped to @@ -532,17 +556,86 @@ we should color these blocks differently *** TODO refine html exportation should use a span class, and should show original source in tool-tip -** STARTED Column (and row) names of tables in R input/output -*** DONE column names - This has been implemented: Automatic on input to R; optional in output. -*** TODO row names - Perhaps add a :rownames header arg. This would be an integer +** TODO LoB: re-implement plotting and analysis functions from org-R + I'll do this soon, now that we things are a bit more settled and we + have column names in R. +** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format + I haven't thought about this properly. Just noting it down. What + Sweave uses is called "R noweb" (.Rnw). + + I found a good description of noweb in the following article (see + the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]). + + I think there are two parts to noweb, the construction of + documentation and the extraction of source-code (with notangle). + + *documentation*: org-mode handles all of our documentation needs in + a manner that I believe is superior to noweb. + + *source extraction* At this point I don't see anyone writing large + applications with 100% of the source code contained in org-babel + files, rather I see org-babel files containing things like + - notes with active code chunks + - interactive tutorials + - requirements documents with code running test suites + - and of course experimental reports with the code to run the + experiment, and perform analysis + + Basically I think the scope of the programs written in org-babel + (at least initially) will be small enough that it wont require the + addition of a tangle type program to extract all of the source code + into a running application. + + On the other hand, since we already have named blocks of source + code which reference other blocks on which they rely, this + shouldn't be too hard to implement either on our own, or possibly + relying on something like noweb/notangle. + +** PROPOSED support for passing paths to files between source blocks +Maybe this should be it's own result type (in addition to scalars and +vectors). The reason being that some source-code blocks (for example +ditaa or anything that results in the creation of a file) may want to +pass a file path back to org-mode which could then be inserted into +the org-mode buffer as a link to the file... + +This would allow for display of images upon export providing +functionality similar to =org-exp-blocks= only in a more general +manner. +** DEFERRED Support rownames and other org babel table features? + + The full org table features are detailed in the manual [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]]. + +*** rownames + Perhaps add a :rownames header arg. This would be an integer (usually 1) which would have the effect of post-processing all the variables created in the R session in the following way: if the integer is j, set the row names to the contents of column j and delete column j. Perhaps it is artificial to allow this integer to take any value other than 1. The default would be nil which would mean no such behaviour. + + Actually I don't know about that. If multiple variables are passed + in, it's not appropriate to alter them all in the same way. The + rownames specification would normally refer to just one of the + variables. For now maybe just say this has to be done in R. E.g. + +#+TBLNAME: sample-sizes + | collection | size | exclude | include | exclude2 | include2 | + |-----------------+------+---------+---------+----------+----------| + | 58C | 2936 | 8 | 2928 | 256 | 2680 | + | MS | 5852 | 771 | 5081 | 771 | 5081 | + | NBS | 2929 | 64 | 2865 | 402 | 2527 | + | POBI | 2717 | 1 | 2716 | 1 | 2716 | + | 58C+MS+NBS+POBI | | | 13590 | | 13004 | +#+TBLFM: @2$4=@2$2 - @2$3::@2$6=@2$2 - @2$5::@3$4=@3$2-@3$3::@3$6=@3$2 - @3$5::@4$4=@4$2 - @4$3::@4$6=@4$2 - @4$5::@5$4=@5$2-@5$3::@5$6=@5$2 - @5$5::@6$4=vsum(@2$4..@5$4)::@6$6=vsum(@2$6..@5$6) + +#+srcname: make-size-table(size=sample-sizes) +#+begin_src R + rownames(size) <- size[,1] + size <- size[,-1] +#+end_src + + *** Old notes [I don't think it's as problematic as this makes out] This is non-trivial, but may be worth doing, in particular to @@ -640,106 +733,6 @@ tabel Another example is in the [[*operations%20in%20on%20tables][grades example]]. -** TODO re-implement helper functions from org-R -*** Initial statement [Eric] - Much of the power of org-R seems to be in it's helper functions for - the quick graphing of tables. Should we try to re-implement these - functions on top of org-babel? - - I'm thinking this may be useful both to add features to org-babel-R and - also to potentially suggest extensions of the framework. For example - one that comes to mind is the ability to treat a source-code block - like a function which accepts arguments and returns results. Actually - this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]). -*** Objectives [Dan] - - We want to provide convenient off-the-shelf actions - (e.g. plotting data) that make use of our new code evaluation - environment but do not require any actual coding. -*** Initial Design proposal [Dan] - - *Input data* will be specified using the same mechanism as :var - references, thus the input data may come from a table, or - another source block, and it is initially available as an elisp - data structure. - - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that - line will apply an *action* to the referenced data. - - *Actions correspond to source blocks*: our library of available - actions will be a library of org-babel source blocks. Thus the - code for executing an action, and the code for dealing with the - output of the action will be the same code as for executing - source blocks in general - - Optionally, the user can have the relevant source block inserted - into the org buffer after the (say) #+BABELDO line. This will - allow the user to fine tune the action by modifying the code - (especially useful for plots). - - So maybe a #+BABELDO line will have header args - - :data (a reference to a table or source code block) - - :action (or should that be :srcname?) which will be something - like :action pie-chart, referring to a source block which will - be executed with the :data referent passed in using a :var arg. - - :showcode or something controlling whether to show the code - -*** Modification to design - I'm implementing this, at least initially, as a new interpreter - named 'babel', which has an empty body. 'babel' blocks take - a :srcname header arg, and look for the source-code block with - that name. They then execute the referenced block, after first - appending their own header args on to the target block's header - args. - - If the target block is in the library of babel (a.o.t. e.g. the - current buffer), then the code in the block will refer to the - input data with a name dictated by convention (e.g. __data__ - (something which is syntactically legal in all languages...). Thus - the babel block will use a :var __data__ = whatever header arg to - reference the data to be plotted. - -*** Current design - This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain - ready-made source blocks designed to carry out useful common tasks. - -** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format - I haven't thought about this properly. Just noting it down. What - Sweave uses is called "R noweb" (.Rnw). - - I found a good description of noweb in the following article (see - the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]). - - I think there are two parts to noweb, the construction of - documentation and the extraction of source-code (with notangle). - - *documentation*: org-mode handles all of our documentation needs in - a manner that I believe is superior to noweb. - - *source extraction* At this point I don't see anyone writing large - applications with 100% of the source code contained in org-babel - files, rather I see org-babel files containing things like - - notes with active code chunks - - interactive tutorials - - requirements documents with code running test suites - - and of course experimental reports with the code to run the - experiment, and perform analysis - - Basically I think the scope of the programs written in org-babel - (at least initially) will be small enough that it wont require the - addition of a tangle type program to extract all of the source code - into a running application. - - On the other hand, since we already have named blocks of source - code which reference other blocks on which they rely, this - shouldn't be too hard to implement either on our own, or possibly - relying on something like noweb/notangle. - -** PROPOSED support for passing paths to files between source blocks -Maybe this should be it's own result type (in addition to scalars and -vectors). The reason being that some source-code blocks (for example -ditaa or anything that results in the creation of a file) may want to -pass a file path back to org-mode which could then be inserted into -the org-mode buffer as a link to the file... - -This would allow for display of images upon export providing -functionality similar to =org-exp-blocks= only in a more general -manner. - ** DEFERRED use textConnection to pass tsv to R? When passing args from the org buffer to R, the following route is used: arg in buffer -> elisp -> tsv on file -> data frame in R. I @@ -932,6 +925,70 @@ to the command if BUFF is not given.) 2) The function is called inside of a =write.table= function call writing the results to a table 3) The table is read using =org-table-import= + +** DONE extensible library of callable source blocks +*** Current design + This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain + ready-made source blocks designed to carry out useful common tasks. +*** Initial statement [Eric] + Much of the power of org-R seems to be in it's helper functions for + the quick graphing of tables. Should we try to re-implement these + functions on top of org-babel? + + I'm thinking this may be useful both to add features to org-babel-R and + also to potentially suggest extensions of the framework. For example + one that comes to mind is the ability to treat a source-code block + like a function which accepts arguments and returns results. Actually + this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]). +*** Objectives [Dan] + - We want to provide convenient off-the-shelf actions + (e.g. plotting data) that make use of our new code evaluation + environment but do not require any actual coding. +*** Initial Design proposal [Dan] + - *Input data* will be specified using the same mechanism as :var + references, thus the input data may come from a table, or + another source block, and it is initially available as an elisp + data structure. + - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that + line will apply an *action* to the referenced data. + - *Actions correspond to source blocks*: our library of available + actions will be a library of org-babel source blocks. Thus the + code for executing an action, and the code for dealing with the + output of the action will be the same code as for executing + source blocks in general + - Optionally, the user can have the relevant source block inserted + into the org buffer after the (say) #+BABELDO line. This will + allow the user to fine tune the action by modifying the code + (especially useful for plots). + - So maybe a #+BABELDO line will have header args + - :data (a reference to a table or source code block) + - :action (or should that be :srcname?) which will be something + like :action pie-chart, referring to a source block which will + be executed with the :data referent passed in using a :var arg. + - :showcode or something controlling whether to show the code + +*** Modification to design + I'm implementing this, at least initially, as a new interpreter + named 'babel', which has an empty body. 'babel' blocks take + a :srcname header arg, and look for the source-code block with + that name. They then execute the referenced block, after first + appending their own header args on to the target block's header + args. + + If the target block is in the library of babel (a.o.t. e.g. the + current buffer), then the code in the block will refer to the + input data with a name dictated by convention (e.g. __data__ + (something which is syntactically legal in all languages...). Thus + the babel block will use a :var __data__ = whatever header arg to + reference the data to be plotted. + +** DONE Column names in R input/output + This has been implemented: Automatic on input to R; optional in + output. Note that this equates column names with the header row in + an org table; whereas org actually has a mechanism whereby a row + with a '!' in the first field defines column names. I have not + attempted to support these org table mechanisms yet. See [[*Support%20rownames%20and%20other%20org%20babel%20table%20features][this + DEFERRED todo item]]. ** DONE use example block for large amounts of stdout output? We're currently `examplizing' with : at the beginning of the line, but should larger amounts of output be in a @@ -2138,8 +2195,90 @@ to specify a file holding the results (see [[* file result types][file result types]]) -* Bugs [22/34] +* Bugs [22/36] +** TODO Default args + This would be good thing to address soon. I'm imagining that + e.g. here, the 'caller' block would return the answer 30. I believe + there's a few issues here: i.e. the naked 'a' without a reference + is not understood; the default arg b=6 is not understood. + +#+srcname: adder(a, b=6) +#+begin_src python +a+b +#+end_src + +#+srcname: caller(var=adder(a=24)) +#+begin_src python +var +#+end_src +** TODO function calls in #+srcname: refs + + My srcname references don't seem to be working for function + calls. This needs fixing. + +#+srcname: called +#+begin_src python +56 +#+end_src + +srcname function call doesn't work for calling a source block +#+srcname: caller(var1=called()) +#+begin_src python +var1 +#+end_src + +They do work for a simple reference +#+srcname: caller(var1=56) +#+begin_src python +var1 +#+end_src + +#+resname: caller +: 56 + +#+srcname: caller2 +#+begin_src python :var var1=called() +var1 +#+end_src + +#+resname: +: 56 +** TODO creeping blank lines + There's still inappropriate addition of blank lines in some circumstances. + + Hmm, it's a bit confusing. It's to do with o-b-remove-result. LoB + removes the entire (#+resname and result) and starts from scratch, + whereas #+begin_src only removes the result. I haven't worked out + what the correct fix is yet. Maybe the right thing to do is to make + sure that those functions (o-b-remove-result et al.) are neutral + with respect to newlines. Sounds easy, but... + + E.g. + +#+begin_src sh +b=5 +#+end_src + + + + Compare the results of +#+lob: python-add(a=5, b=17) + +#+resname: python-add(a=5, b=17) +: 22 +-------------------------------- + +#+begin_src python +23 +#+end_src + +#+resname: +: 23 +--------------------- + ** TODO avoid stripping whitespace from output when :results output + This may be partly solved by using o-b-chomp rather than o-b-trim + in the o-b-LANG-evaluate functions. ** TODO problem with newlines in output when :results value #+begin_src python :results value '\n'.join(map(str, range(4))) @@ -2147,6 +2286,87 @@ to specify a file holding the results #+resname: : 0 + +Whereas I was hoping for + +| 0 | +| 1 | +| 2 | +| 3 | + +This is some sort of non-printing char / quoting issue I think. Note +that + +#+begin_src python :results value +'\\n'.join(map(str, range(4))) +#+end_src + +#+resname: +: 0\n1\n2\n3 + +Also, note that +#+begin_src python :results output +print('\n'.join(map(str, range(4)))) +#+end_src + +#+resname: +: 0 +: 1 +: 2 +: 3 + +*** collapsing consecutive newlines in string output + + This is an example of the same bug + +#+srcname: multi-line-string-output +#+begin_src ruby :results output +"the first line ends here + + + and this is the second one + +even a third" +#+end_src + +This doesn't produce anything at all now. I believe that's because +I've changed things so that :results output really does *not* get the +value of the block, only the STDOUT. So if we add a print statement +this works OK. + +#+srcname: multi-line-string-output +#+begin_src ruby :results output +print "the first line ends here + + + and this is the second one + +even a third" +#+end_src + +#+resname: +: the first line ends here +: +: +: and this is the second one +: +: even a third + +However, the behaviour with :results value is wrong + +#+srcname: multi-line-string-value +#+begin_src ruby +"the first line ends here + + + and this is the second one + +even a third" +#+end_src + +#+resname: +: 0 + ** TODO prompt characters appearing in output with R #+begin_src R :session *R* :results output x <- 6 @@ -2179,20 +2399,6 @@ for example #+end_src #+resname: this-doesn't-match-orgtbl - -** TODO collapsing consecutive newlines in string output - -#+srcname: multi-line-string-output -#+begin_src ruby :results output -"the first line ends here - - - and this is the second one - -even a third" -#+end_src - -#+resname: multi-line-string-output ** PROPOSED external shell execution can't isolate return values I have no idea how to do this as of yet. The result is that when @@ -2221,33 +2427,6 @@ the same for the other languages. [Dan] not org-babel-ruby-evaluate ** TODO use new merge function [[file:lisp/org-babel-ref.el::t%20nil%20org%20combine%20plists%20args%20nil][here]]? And at other occurrences of org-combine-plists? -** TODO creeping blank lines - There's still inappropriate addition of blank lines in some circumstances. E.g. - -#+begin_src sh -b=5 -#+end_src - - - Compare the results of -#+lob: python-add(a=5, b=17) - -#+resname: python-add(a=5, b=17) -: 22 --------------------------------- - -#+begin_src python -23 -#+end_src - -#+resname: -: 23 ---------------------- - - Hmm, it's a bit confusing. I think it's to do with the fact that - LoB removes the entire (#+resname and result) and starts from - scratch, whereas #+begin_src only removes the result. I haven't - worked out what the correct fix is yet. ** TODO LoB is not populated on startup org-babel-library-of-babel is nil for me on startup. I have to evaluate the [[file:lisp/org-babel-lob.el::][org-babel-lob-ingest]] line manually. From 162b52272844ff702cdb152310f76215c709ce34 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Sun, 19 Jul 2009 14:29:10 -0400 Subject: [PATCH 3/8] Fixing srcname function call regexps. But still some improvement of this to be done (see bugs section). --- lisp/org-babel.el | 5 +-- org-babel.org | 80 ++++++++++++++++++++++++++++------------------- 2 files changed, 51 insertions(+), 34 deletions(-) diff --git a/lisp/org-babel.el b/lisp/org-babel.el index a2cc4ce44..7816df2e9 100644 --- a/lisp/org-babel.el +++ b/lisp/org-babel.el @@ -232,7 +232,7 @@ the word 'call'." (goto-char head) (if (save-excursion (forward-line -1) - (looking-at "#\\+srcname:[ \f\t\n\r\v]*\\([^ \f\t\n\r\v]+\\)\(\\(.*\\)\)")) + (looking-at "#\\+srcname:[ \f\t\n\r\v]*\\([^ ()\f\t\n\r\v]+\\)\(\\(.*\\)\)")) (org-babel-clean-text-properties (match-string 1))))))) (defun org-babel-get-src-block-info () @@ -360,7 +360,8 @@ according to org-babel-named-src-block-regexp." buffer or nil if no such result exists." (save-excursion (goto-char (point-min)) - (when (re-search-forward (concat "#\\+resname:[ \t]*" (regexp-quote name)) nil t) + (when (re-search-forward ;; ellow end-of-buffer in following regexp? + (concat "#\\+resname:[ \t]*" (regexp-quote name) "[ \t\n\f\v\r]") nil t) (move-beginning-of-line 1) (point)))) (defun org-babel-where-is-src-block-result () diff --git a/org-babel.org b/org-babel.org index b0a94b57b..dd0df9597 100644 --- a/org-babel.org +++ b/org-babel.org @@ -2201,7 +2201,7 @@ to specify a file holding the results (see [[* file result types][file result types]]) -* Bugs [22/36] +* Bugs [23/36] ** TODO Default args This would be good thing to address soon. I'm imagining that e.g. here, the 'caller' block would return the answer 30. I believe @@ -2217,38 +2217,11 @@ a+b #+begin_src python var #+end_src -** TODO function calls in #+srcname: refs - - My srcname references don't seem to be working for function - calls. This needs fixing. - -#+srcname: called -#+begin_src python -56 -#+end_src +** TODO allow srcname to omit function call parentheses + Someone needs to revisit those regexps. Is there an argument for + moving some of the regexps used to match function calls into + defvars? (i.e. in o-b.el and o-b-ref.el) -srcname function call doesn't work for calling a source block -#+srcname: caller(var1=called()) -#+begin_src python -var1 -#+end_src - -They do work for a simple reference -#+srcname: caller(var1=56) -#+begin_src python -var1 -#+end_src - -#+resname: caller -: 56 - -#+srcname: caller2 -#+begin_src python :var var1=called() -var1 -#+end_src - -#+resname: -: 56 ** TODO creeping blank lines There's still inappropriate addition of blank lines in some circumstances. @@ -2464,6 +2437,48 @@ the same for the other languages. [Dan] the user's regular emacs init. I can't think of a way for us to set this automatically, and we are SOL without a regexp to match the prompt. +** DONE function calls in #+srcname: refs + + My srcname references don't seem to be working for function + calls. This needs fixing. + +#+srcname: called() +#+begin_src python +59 +#+end_src + +srcname function call doesn't work for calling a source block +#+srcname: caller(var1=called()) +#+begin_src python +var1 +#+end_src + +#+resname: caller +: 59 + + + + + + +They do work for a simple reference +#+srcname: caller2(var1=56) +#+begin_src python +var1 +#+end_src + +#+resname: caller2 +: 59 + + +and they do work for :var header arg +#+srcname: caller3 +#+begin_src python :var var1=called() +var1 +#+end_src + +#+resname: +: 58 ** DONE LoB: with output to buffer, not working in buffers other than library-of-babel.org *** Initial report I haven't fixed this yet. org-babel-ref-resolve-reference moves @@ -2916,6 +2931,7 @@ of these tests may fail. #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5)) #+TBLFM: $5=""::$6="" + The second TBLFM line (followed by replacing '[]' with '') can be used to blank out the table results, in the absence of a better method. From 7ac2e0d45f89e87a5accf2ddfb7629c632860d89 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Sun, 19 Jul 2009 17:35:49 -0400 Subject: [PATCH 4/8] Using o-b-chomp rather than o-b-trim on R output. This keeps columns of STDOUT correctly aligned. --- lisp/langs/org-babel-R.el | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lisp/langs/org-babel-R.el b/lisp/langs/org-babel-R.el index c2ca67936..953c4ac62 100644 --- a/lisp/langs/org-babel-R.el +++ b/lisp/langs/org-babel-R.el @@ -137,9 +137,9 @@ last statement in BODY, as elisp." (if (string-match comint-prompt-regexp el) (substring el (match-end 0)) el)))) - (mapcar #'org-babel-trim raw)))))) + (mapcar #'org-babel-chomp raw)))))) (case result-type - (output (org-babel-chomp (mapconcat #'identity results "\n"))) + (output (mapconcat #'identity results "\n")) (value (org-babel-R-process-value-result (org-babel-import-elisp-from-file tmp-file) column-names-p))))))) From 2b66de1c1bc9e966cfa7c8711e0699c622ad64e1 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Sun, 19 Jul 2009 17:45:00 -0400 Subject: [PATCH 5/8] Reorganising R evaluation code. With this change we avoid messing about extracting the output from the comint buffer in the :results value case (the value has already been written to file). --- lisp/langs/org-babel-R.el | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/lisp/langs/org-babel-R.el b/lisp/langs/org-babel-R.el index c2ca67936..0a0234578 100644 --- a/lisp/langs/org-babel-R.el +++ b/lisp/langs/org-babel-R.el @@ -122,26 +122,26 @@ last statement in BODY, as elisp." (full-body (mapconcat #'org-babel-chomp (list body last-value-eval org-babel-R-eoe-indicator) "\n")) (raw (org-babel-comint-with-output buffer org-babel-R-eoe-output nil - (insert full-body) (inferior-ess-send-input))) - (results - (let ((broke nil)) - (delete - nil - (mapcar (lambda (el) - (if (or broke - (and (string-match (regexp-quote org-babel-R-eoe-output) - el) (setq broke t))) - nil - (if (= (length el) 0) - nil - (if (string-match comint-prompt-regexp el) - (substring el (match-end 0)) - el)))) - (mapcar #'org-babel-trim raw)))))) + (insert full-body) (inferior-ess-send-input))) broke results) (case result-type - (output (org-babel-chomp (mapconcat #'identity results "\n"))) (value (org-babel-R-process-value-result - (org-babel-import-elisp-from-file tmp-file) column-names-p))))))) + (org-babel-import-elisp-from-file tmp-file) column-names-p)) + (output + (setq broke nil results + (delete + nil + (mapcar (lambda (el) + (if (or broke + (and (string-match (regexp-quote org-babel-R-eoe-output) el) + (setq broke t))) + nil + (if (= (length el) 0) + nil + (if (string-match comint-prompt-regexp el) + (substring el (match-end 0)) + el)))) + (mapcar #'org-babel-chomp raw)))) + (mapconcat #'identity results "\n"))))))) (defun org-babel-R-process-value-result (result column-names-p) "R-specific processing of return value prior to return to org-babel. From b766b094a081cdd1c14465f02e4fc1ea2078a858 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Sun, 19 Jul 2009 18:03:33 -0400 Subject: [PATCH 6/8] Using temp function definition to tidy R evaluation code. --- lisp/langs/org-babel-R.el | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/lisp/langs/org-babel-R.el b/lisp/langs/org-babel-R.el index 0a0234578..4e0b2a858 100644 --- a/lisp/langs/org-babel-R.el +++ b/lisp/langs/org-babel-R.el @@ -126,22 +126,21 @@ last statement in BODY, as elisp." (case result-type (value (org-babel-R-process-value-result (org-babel-import-elisp-from-file tmp-file) column-names-p)) - (output - (setq broke nil results - (delete - nil - (mapcar (lambda (el) - (if (or broke - (and (string-match (regexp-quote org-babel-R-eoe-output) el) - (setq broke t))) - nil - (if (= (length el) 0) - nil - (if (string-match comint-prompt-regexp el) - (substring el (match-end 0)) - el)))) - (mapcar #'org-babel-chomp raw)))) - (mapconcat #'identity results "\n"))))))) + (output + (flet ((extractor + (el) + (if (or broke + (and (string-match (regexp-quote org-babel-R-eoe-output) el) + (setq broke t))) + nil + (if (= (length el) 0) + nil + (if (string-match comint-prompt-regexp el) + (substring el (match-end 0)) + el))))) + (mapconcat + #'identity + (delete nil (mapcar #'extractor (mapcar #'org-babel-chomp raw))) "\n")))))))) (defun org-babel-R-process-value-result (result column-names-p) "R-specific processing of return value prior to return to org-babel. From 5a5a4253c6bb98d6f3e241145fdf681919c5c2d7 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Mon, 20 Jul 2009 01:24:33 -0400 Subject: [PATCH 7/8] Some notes on possibly adding a recursive evaluation mechanism. Currently we can't have nested function calls, and I think that a general fix for that would require going the standard route of constructing a parse tree and evaluating it recursively. --- org-babel.org | 144 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 121 insertions(+), 23 deletions(-) diff --git a/org-babel.org b/org-babel.org index dd0df9597..3995f4a3f 100644 --- a/org-babel.org +++ b/org-babel.org @@ -207,7 +207,121 @@ would then be [[#sandbox][the sandbox]]. #+end_src -* Tasks [35/55] +* Tasks [35/57] +** PROPOSED implement fully recursive evaluation machinery + The current parser / evaluator fails with greater levels of nested + function block calls (example below). If we want to overcome this I + think we'd have to redesign some of the evaluation + mechanism. Seeing as we are also facing issues like dealing with + default argument values, and seeing as we now know how we want the + library of babel to behave in addition to the source blocks, now + might be a good time to think about this. It would be nice to do + the full thing at some point, but otoh we may not consider it a + massive priority. + + AIui, there are two stages: (i) construct a parse tree, and (ii) + evaluate it and return the value at the root. In the parse tree + each node represents an unevaluated value (either a literal value + or a reference). Node v may have descendent nodes, which represent + values upon which node v's evaluation depends. Once that tree is + constructed, then we evaluate the nodes from the tips towards the + root (a post-order traversal). + + [This would also provide a solution for concatenating the STDOUTs + of called blocks, which is a [[*allow%20output%20mode%20to%20return%20stdout%20as%20value][task below]]; we concatenate them in + whatever order the traversal is done in.] + + In addition to the variable references (i.e. daughter nodes), each + node would contain the information needed to evaluate that node + (e.g. lang body). Then we would pass a function postorder over the + tree which would call o-b-execute-src-block at each node, finally + returning the value at the root. + + Fwiw I made a very tentative small start at stubbing this out in + org-babel-call.el in the 'evaluation' branch. And I've made a start + at sketching a parsing algorithm below. + +*** Parse tree algorithm + Seeing as we're just trying to parse a string like + f(a=1,b=g(c=2,d=3)) it shouldn't be too hard. But of course there + are 'proper' parsers written in elisp out there, + e.g. [[http://cedet.sourceforge.net/semantic.shtml][Semantic]]. Perhaps we can find what we need -- our syntax is + pretty much the same as python and R isn't it? + + Or, a complete hack, but maybe it would be we easy to transform it + to XML and then parse that with some existing tool? + + But if we're doing it ourselves, something very vaguely like this? + (I'm sure there're lots of problems with this) + +#+srcname: org-babel-call-parse(call) +#+begin_src python + ## we are currently reading a reference name: the name of the root function + whereami = "refname" + node = root = Node() + for c in call_string: + if c == '(': + varnum = 0 + whereami = "varname" # now we're reading a variable name + if c == '=': + new = Node() + node.daughters = [node.daughters, new] + node = new + whereami = "refname" + if c == ',': + whereami = "varname" + varnum += 1 + elif c == '(': + node = node.parent + elif c == ' ': + pass + else: + if whereami = "varname": + node.varnames[varnum] += c + elif whereami = "refname": + node.name += c +#+end_src + +*** Example that fails + +#+srcname: adder(a=0, b=0) +#+begin_src python +a+b +#+end_src + +#+srcname: one() +#+begin_src python :results silent +1 +#+end_src + +This works +#+srcname: level-one-nesting +#+begin_src python :var arg=adder(a=one(),b=one()) +arg +#+end_src + +But this doesn't +#+srcname: level-one-nesting +#+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one())) +arg +#+end_src + +** PROPOSED Default args + This would be good thing to address soon. I'm imagining that + e.g. here, the 'caller' block would return the answer 30. I believe + there's a few issues here: i.e. the naked 'a' without a reference + is not understood; the default arg b=6 is not understood. + +#+srcname: adder(a, b=6) +#+begin_src python +a+b +#+end_src + +#+srcname: caller(var=adder(a=24)) +#+begin_src python +var +#+end_src + ** PROPOSED allow `anonymous' function block with function call args? My question here is simply whether we're going to allow #+begin_src python(arg=ref) @@ -358,7 +472,7 @@ org-mode core org-src-mode would be better. **** DONE Changed minor mode name and added hooks -**** DEFERRED a hook called when the src edit buffer is created +**** DONE a hook called when the src edit buffer is created This should be implemented in the org-mode core ** TODO resolve references to other org buffers/files @@ -459,7 +573,6 @@ This could also act reasonably with other results types... - file :: use org-open-at-point to open the file - scalar :: open results unquoted in a new buffer - tabular :: export the table to a new buffer and open that buffer - ** TODO Finalise behaviour regarding vector/scalar output *** DONE Stop spaces causing vector output This simple example of multilingual chaining produces vector output if @@ -2201,22 +2314,7 @@ to specify a file holding the results (see [[* file result types][file result types]]) -* Bugs [23/36] -** TODO Default args - This would be good thing to address soon. I'm imagining that - e.g. here, the 'caller' block would return the answer 30. I believe - there's a few issues here: i.e. the naked 'a' without a reference - is not understood; the default arg b=6 is not understood. - -#+srcname: adder(a, b=6) -#+begin_src python -a+b -#+end_src - -#+srcname: caller(var=adder(a=24)) -#+begin_src python -var -#+end_src +* Bugs [24/36] ** TODO allow srcname to omit function call parentheses Someone needs to revisit those regexps. Is there an argument for moving some of the regexps used to match function calls into @@ -2254,10 +2352,6 @@ b=5 #+resname: : 23 --------------------- - -** TODO avoid stripping whitespace from output when :results output - This may be partly solved by using o-b-chomp rather than o-b-trim - in the o-b-LANG-evaluate functions. ** TODO problem with newlines in output when :results value #+begin_src python :results value '\n'.join(map(str, range(4))) @@ -2409,6 +2503,10 @@ the same for the other languages. [Dan] ** TODO LoB is not populated on startup org-babel-library-of-babel is nil for me on startup. I have to evaluate the [[file:lisp/org-babel-lob.el::][org-babel-lob-ingest]] line manually. + +** DONE avoid stripping whitespace from output when :results output + This may be partly solved by using o-b-chomp rather than o-b-trim + in the o-b-LANG-evaluate functions. ** DEFERRED weird escaped characters in shell prompt break shell evaluation E.g. this doesn't work. Should the shell sessions set a sane prompt when they start up? Or is it a question of altering From 4589aacb5112b808f9c3c345f2f30d7b29c610f9 Mon Sep 17 00:00:00 2001 From: Dan Davison Date: Mon, 20 Jul 2009 01:33:40 -0400 Subject: [PATCH 8/8] A couple of fixes to sketched-out parsing algorithm. --- org-babel.org | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/org-babel.org b/org-babel.org index 3995f4a3f..ed13af433 100644 --- a/org-babel.org +++ b/org-babel.org @@ -266,12 +266,13 @@ would then be [[#sandbox][the sandbox]]. if c == '=': new = Node() node.daughters = [node.daughters, new] + new.parent = node node = new whereami = "refname" if c == ',': whereami = "varname" varnum += 1 - elif c == '(': + elif c == ')': node = node.parent elif c == ' ': pass