* lisp/org-element.el (org-element-with-disabled-cache): New macro.
(org-element-greater-elements): Add new org-data element. It
functions like a virtual headline containing the whole buffer. The
org-data properties are like headlie properties, but according to the
top-level drawer. org-data's category is the buffer's category as
defined by top-level property drawer, #+CATEGORY keyworsd, and the
buffer file name.
(org-element--cache-element-properties, org-element-set-element): New
variable containing properties to be transferred when updating changed
element in cache in `org-element-set-element'.
(org-element--get-node-properties): Allow parsing node propreties in
top-level drawer when new optional argument is passed. Respect
PROPERTY+ syntax.
(org-element--get-global-node-properties): New function. It returns
node properties for top-level property drawer.
(org-element-org-data-parser, org-element-org-data-interpreter):
Implement the new org-data element.
(org-element-headline-parser, org-element-section-parser): Add new
:robust-begin and :robust-end
properties delimiting safe changes that do not modify headline
element.
(org-element--list-struct): Fix cache update when adding a headline
inside list.
(org-element--current-element): Implement cache support. Record
parsing mode (:mode) and parsing granularity (:granularity) in the
element properties.
(org-element-parse-buffer, org-element--next-mode): Support new
org-data element.
(org-element--parse-elements): Record parsing granularity in the
returned tree
(org-element-use-cache): Enable cache by default.
(org-element-cache-persistent): New variable controlling cache
persistance across sessions. Enabled by default.
(org-element--cache-self-verify,
org-element--cache-self-verify-frequency,
org-element--cache-diagnostics, org-element--cache-map-statistics,
org-element--cache-map-statistics-threshold,
org-element--cache-diagnostics-level,
org-element--cache-diagnostics-ring,
org-element--cache-diagnostics-ring-size): New variables controlling
cache diagnostics and self-diagnostics. Greatly simplifies cache
debugging.
(org-element--cache, org-element--cache-sync-requests,
org-element--cache-sync-timer): Make cache buffer-local by default.
(org-element--headline-cache): Implement separate cache storing only
headlines and inlinetasks.
(org-element--cache-size, org-element--headline-cache-size): New
variables containing cache sizes. This is much faster than
`avl-tree-size'.
(org-element--cache-sync-requests): Update docstring explaning the
request list structure.
(org-element--cache-sync-keys-value): New variable replacing
`org-element--cache-sync-keys' hash table. The hash table was not
reliable because it was using elements as keys. Upon any cached
element update/shift, the keys were invalidated making cache ordering
incorrect and breaking the cache badly. Now, the cache keys are
stored as :org-element--cache-sync-key element property and the new
variable stores marker value indicating the current sync request
cycle. See `org-element--cache-key' for more details.
(org-element--cache-change-tic): New variable controlling buffer
modification count that is registered in cache. This variable allows
catching "stealth" edits.
(org-element--cache-non-modifying-commands): New variable listing
commands that will not be slown down if we fill cache on the fly.
(org-element--request-key, org-element--request-beg,
org-element--request-end, org-element--request-offset,
org-element--request-parent, org-element--request-phase): New macros.
They improve code readability (especially when using nameless-mode).
(org-element--format-element, org-element--cache-log-message,
org-element--cache-warn): New macros implementing generic logging
functionality.
(org-element--cache-key): Add section and org-data element support.
Change cache key storage from hash map to :org-element--cache-sync-key
element property + `org-element--cache-sync-keys-value'. We use the
latter to group all the cache keys during a single cache request
sequence. Once sync request is fully complete, the
`org-element--cache-sync-keys-value' is updated making all the old
sync keys obsolete (they will still be store as element properties).
(org-element--headline-cache-root): New function returning headline
cache root.
(org-element--cache-active-p): Prevent cache updates when
`inhibit-modification-hooks' is non-nil, unless non-nil optional
argument is provided.
(org-element--cache-find): Share cache between indirect buffers and
the base buffer. We have to do it because after-change hooks for
indirect buffer are not called in the base buffer and vice versa. Add
support for section and org-data elements.
(org-element--cache-put): Implement new approach for cache key
storage. Add diagnostics. Indicate cached elements using :cached
element property. Support cache size calculation.
(org-element--cache-remove): Invalidate parent contents when removing
element. Support cache size calculation. Detect cache corruption due
to misordered elements.
(org-element--cache-shift-positions): Support :robust-begin and
:robust-end element properties.
(org-element--cache-sync): Add diagnostics. Add detailed comments.
Prevent slowdown when large cache chunks need to be deleted forcing
O(N) complexity cutoff. In phase 2, fix cases when next request
contains deleted cache key. In phase 2, fix scenario when newly
inserted element intersects with existing elements in cache. In phase
2, detect obsolete parents removed from cache.
(org-element--open-end-p): New function checking if an element can
have blank lines right after its :contents-end.
(org-element--parse-to): Do not alter match data. Process complex
parsing mode changes correctly. Support headlines in cache. Support
org-data parsing. Add detailed comments. Add diagnostics.
(org-element--cache-sensitive-re): Make list lines sensitive.
(org-element--cache-change-warning): Update docstring. Now, the
variable can have t, nil, and number values. Numbers are used to
provide more details about changed headlines.
(org-element--cache-before-change, org-element--cache-after-change):
Handle headline hierarchy. Properly handle cache in indirect
buffers.
(org-element--cache-after-change): Update docstring clarifying the
return values. Add special handling for headline and org-data
elements updating them in-place instead of removing together with the
whole contents when possible. Use :robust-begin/:robust-end element
properties to detect robust changes.
(org-element--cache-submit-request): Add detailed comments. Correctly
handle cache in indirect buffers. Delegate element modifications to
`org-element--cache-for-removal'.
(org-element--cache-verify-element): New function for cache
self-verification.
(org-element--cache-persist-before-write,
org-element--cache-persist-before-read,
org-element--cache-persist-after-read): Implement cache persistance.
(org-element-cache-reset): Correctly handle cache in indirect
buffers. Support cache persistance. Support new cache size
calculation and new cache key schema.
(org-element-cache-map): New function analagous to `org-element-map',
but much faster. The function overperforms org-ql written by Adam
Porter aka alphapapa [1] and reuses some ideas from there (namely,
fast element skipping via regexps).
[1] https://github.com/alphapapa/org-ql/
(org-element-at-point): The returned elements are now guaranteed to
have correct parents up to org-data. New optional argument
CACHED-ONLY limits element search to current cache---if element is not
in cache and current command is not in cache
`org-element--cache-non-modifying-commands', the cache is not updated
and the function returns nil. Also, support cache verification.
(org-element-at-point-no-context): New function. It is analogous of
older `org-element-at-point' with no guarantee that :parent properties
are correct beyond direct parent heading. This function does not
update cache and can be useful when cache updates should be avoided
for performance reasons.
* lisp/ob-core.el (org-babel-where-is-src-block-result): Support
section and org-data elements in cache.
* lisp/org-macro.el (org-macro-replace-all,
org-macro--find-keyword-value): Support org-element-cache.
* lisp/org-table.el (orgtbl-to-generic): Support org-element-cache.
* lisp/org.el (org-mode): Add cache persistance.
(org-up-element): Preserve old behaviour when error is returned for
section and org-data element.
*
testing/lisp/test-org-archive.el (test-org-archive/update-status-cookie):
Fix test when cache is active.
* testing/lisp/test-org-colview.el (test-org-colview/columns-update):
Fix test.
* testing/lisp/test-org-element.el (test-org-element/extract-element):
Add suport for new org-data element.
* testing/lisp/test-org-element.el (test-org-element/parent-property):
Fix equality check. Parents returned by cache and `org-element-map'
may not be `eq' now. Just `equal'.
* testing/lisp/test-org-element.el (test-org-element/context): Support
section and headline parents.
* lisp/ob-core.el (org-babel-eval-headers): Fix elements ordering.
(org-babel-merge-params): Tiny refactoring.
This fixes the test `test-ob-lob/call-with-header-arguments' that
fails after 9f1507bdd.
* lisp/ob-core.el (org-babel-default-header-args): Document ability to
use closures.
(org-babel-eval-headers): New function to generate header arguments,
which adds the ability to evaluate closures during source block
execution or export.
(org-babel-merge-params): Only evaluate closures when we have our
final list of headers.
* lisp/ob-core.el (org-babel-disassemble-tables): Avoid length> for
compatibility with Emacs versions before 28.
length> and friends were added to Emacs in 0f790464d5 (Add new
predicates for sequence lengths, 2020-12-27), which hasn't yet made it
into a release.
* lisp/ob-core.el (org-babel-disassemble-tables,
org-babel-gnuplot-process-vars): Use `proper-list-p' check instead of
`listp'.
(org-babel-disassemble-tables): Check list length before trying to
remove hline from a table assignment.
* testing/lisp/test-ob.el (test-ob/cons-cell-as-variable): Add the
test.
* lisp/ob-core.el (org-babel-default-header-args): Document ability to
use functions.
(eval-default-headers): New function to generate default header
arguments, which adds the ability to evaluate function arguments at
runtime.
(org-babel-get-src-block-info): Use new header argument evaluate
function when retreiving src block info.
* lisp/ob-exp.el (org-babel-exp-src-block): Must use new
eval-default-headers when exporting as well.
The closures are evaluated at runtime.
The commit d4b739fbf2 added the "none" header argument but didn't
add it to the `org-babel-insert-result' documentation string. Put it
in the doc string so that users can discover it.
Signed-off-by: Yasushi SHOJI <yashi@spacecubics.com>
* lisp/ob-core.el (org-babel-expand-noweb-references): Fix order when
searching for named babel blocks so that blocks in the current buffer
are always found first. This fixes a bug where stale versions of
blocks that have been ingested into the library of babel were being
preferentially tangled instead of newly modified versions from the
current buffer.
* lisp/ob-core.el (org-babel-edit-distance): Rename function
`org-string-distance' and move it to "org-compat.el".
(org-babel-check-src-block): Use `org-string-distance' instead.
* lisp/org-compat.el: Add `org-string-distance' for a future
deprecation in favor of `string-distance'.
* lisp/ob-core.el (org-babel-result-end): Alloow the result to end
into a LaTeX environment.
By adding latex-enviroment to the possible results types, a much
richer experience is possible for backends that can produce
LaTeX-formatted maths as their results (for example: Calc, Julia with
Latexify, Python with SymPy, Octave with symbolic). This is superior
to a LaTeX export block as LaTeX environments can be (1) rendered
inline, and (2) better used in non-LaTeX export formats (e.g. HTML
with MathJax). Without this, backends can output a LaTeX environment,
but new results will be added in front instead of replacing the
environment.
Porting from Emacs's master to Org's maint branch has been on hold
leading up to the Emacs 27.2 release to avoid any required fixup syncs
carrying those commits into the emacs-27 branch. This merge brings
those changes into master. The km/from-emacs-master branch should be
merged to maint when Emacs 27.2 has been released (assuming Org 9.5
hasn't been released, in which case maint will be tracking 9.5.x and
already include these changes).
* lisp/ob-comint.el (org-babel-comint-in-buffer)
(org-babel-comint-with-output): Use `declare`.
* lisp/ob-core.el (org-babel-map-src-blocks): Use `declare`.
(org-babel-result-cond): Simplify edebug spec.
* lisp/org-clock.el (org-with-clock-position, org-with-clock):
* lisp/org-agenda.el (org-agenda-with-point-at-orig-entry):
* lisp/ob-tangle.el (org-babel-with-temp-filebuffer): Use `declare`.
Use `declare` instead of `def-edebug-spec` in most places
f8dbefbaa59bb17dd4a2dfa4d9ff560c46785792
Stefan Monnier
Fri Feb 12 16:08:01 2021 -0500
* lisp/ob-core.el (org-babel--string-to-number): Put hyphen last in
alternative.
* lisp/org-agenda.el (org-agenda-filter): Escape '+' correctly.
Follow good regexp practice
46394dff7f01e7fe4af06a6c344e151af5c3eef4
Mattias Engdegård
Fri Dec 18 14:35:09 2020 +0100
* doc/org-manual.org (Type): Document empty vector argument for
file-desc.
* etc/ORG-NEWS (New argument for ~file-desc~ babel header): Add entry
to NEWS.
* lisp/ob-core.el (org-babel--file-desc): Add new function to evaluate
file description value.
(org-babel-execute-src-block): Correctly evaluate file description
when executing src block.
(org-babel-insert-result): Correctly evaluate file description value
when inserting the result of src block execution into the buffer.
* testing/lisp/test-ob.el (test-ob/file-desc-header-argument): Add
test case for new behavior.
* lisp/ob-core.el (org-babel-expand-noweb-references): Don't anchor
noweb regexp at start of line to allow multiple matches per line.
* testing/lisp/test-ob-tangle.el (ob-tangle/multiple-noweb-in-line):
Add test.
This fixes a regression introduced by c1aed9f80 (ob-core: Refactor
`org-babel-expand-noweb-references', 2020-01-12), which was part of
the 9.4 release.
Reported-by: Tom Gillespie <tgbugs@gmail.com>
Ref: https://orgmode.org/list/CA+G3_PO2yO1jMMpdrkc39BGQQ2eU5X4FzTEJVotjDJo-50dsqQ@mail.gmail.com
* lisp/ob-core.el (org-babel--expand-body): New function.
(org-babel-check-confirm-evaluate):
(org-babel-execute-src-block): Use org-babel--expand-body.
As of 727c6d7fe (lisp/ob-core.el: org-babel-check-confirm-evaluate
strip coderefs, 2020-09-04), org-babel-check-confirm-evaluate and
org-babel-execute-src-block use the same logic to expand the body and
strip coderefs. Move that code to a helper function.
This patch is functionally equivalent to the patch originally proposed
by Tom Gillespie at
https://orgmode.org/list/CA+G3_PNi3uMvBiWgBdKuC3C6VJt1T1j-RKH43LRqYbr+4NS8ZA@mail.gmail.com
* lisp/ob-core.el (org-babel-check-confirm-evaluate): strip coderefs
before passing the body of the block to org-confirm-babel-evaluate
using the same let block as is used in org-babel-execute-src-block.
* lisp/ob-core.el (org-babel-result-to-file): Use
`buffer-base-buffer' so that `buffer-file-name' returns correct
results in indirect buffers too.
See <https://orgmode.org/list/87eeos3w7v.fsf@localhost>
TINYCHANGE
* lisp/ob-core.el (org-babel-import-elisp-from-file): Don't try to
convert empty file to a table.
* testing/lisp/test-ob.el (test-ob/import-elisp-from-file): Add tests.
If org-babel-import-elisp-from-file is called with an empty file
(which many ob- libraries do when there are no results), feeding that
to org-table-import leads to a beginning-of-buffer error. Before
14878f3f9 (ob-core: Display warning on failure to read results,
2020-05-21), this error was hard to notice because, after catching it,
it was reported as a quickly buried message. After that commit, it is
displayed as a warning, which is not appropriate for the common and
unproblematic case of empty results.
Avoid the warning by only doing the table conversion if the file has
content. Another option would be to do the table conversion but add a
beginning-of-buffer arm to the surrounding condition-case. However,
that risks swallowing other sources of that error.
Reported-by: Colin Baxter <m43cap@yandex.com>
<https://orgmode.org/list/878se3nhbj.fsf@yandex.com>
* ob-core.el (org-babel-execute-src-block): Source code block header
argument `:file-mode' can set file permissions if `:file' argument is
provided.
(org-babel-common-header-args-w-values): Add `:file-mode' to common
header arguments.
TINYCHANGE
* lisp/ob-core.el (org-babel-check-confirm-evaluate): Expand noweb
references in the body passed to org-confirm-babel-evaluate.
* testing/lisp/test-ob.el (test-ob/check-eval-noweb-expanded): New
test.
When noweb references are set to be expanded for evaluation,
org-confirm-babel-evaluate should receive the expanded body so that it
can make its decision based on what actually will be evaluated
(without resorting to calling org-babel-get-src-block-info itself).
Note that this results in up to three calls to
org-babel-expand-noweb-references. Possible ways to avoid this are
discussed at the thread referenced below.
Reported-by: Tom Gillespie <tgbugs@gmail.com>
https://orgmode.org/list/CA+G3_PNi3uMvBiWgBdKuC3C6VJt1T1j-RKH43LRqYbr+4NS8ZA@mail.gmail.com/
* lisp/ob-core.el (org-babel-import-elisp-from-file): Show handled
errors with display-warning rather than a message because the latter
is quickly overridden by subsequent messages, making it difficult if
not impossible for the user to spot.
The scope of the save-window-excursion call would need to be reduced
for the display-warning buffer to be shown, but nothing appears to
change the window configuration, so just drop the
save-window-excursion call.
Reported-by: Greg Minshall <minshall@umich.edu>
<2449663.1588516024@apollo2.minshall.org>
* lisp/ob-core.el (org-babel-current-result-hash):
(org-babel-hide-hash):
(org-babel-hide-all-hashes):
(org-babel-result-hide-all):
(org-babel-hide-result-toggle-maybe):
(org-babel-hide-result-toggle):
(org-babel--clear-results-maybe):
(org-babel-remove-result): Bind `case-fold-search' to t before looking
for `org-babel-result-regexp'.
* lisp/ob-core.el (org-babel-remove-temporary-directory):
Use (rx (or (not ".") "...")), translated into "[^.]\\|\\.\\.\\.", to
match anything but "." and "..".
Fix edge case errors in filename-matching regexps
4acdd7fe58ae9f94102afeca67b0383141d597da
Mattias Engdegård
Tue Apr 14 12:25:16 2020 +0200
Note(km): Replaced bare "." with (any ".") for Emacs < 27
compatibility. 4acdd7fe5 was on the emacs-27 branch, but a commit on
their master (d5a7df8c0) will supersede this.
These both show up when compiling files separately (e.g., 'make
single').
ob-core already assumes ob-tangle is loaded based on declared
functions, presumably because ob.el is given responsibility for
collectively loading these pieces. org.el loads org-attach within the
body of a function before calling org-attach-dir.
* lisp/ob-core.el (org-babel-result-end): Handle results in special
blocks.
* testing/lisp/test-ob.el (test-ob/replace-special-block-result): New
test.
Make regexps smaller and faster by removing terms that are superfluous
by virtue of standing next to another term that matches more. See
https://lists.gnu.org/archive/html/emacs-devel/2020-01/msg00949.html
for details.
* lisp/ob-core.el (org-babel-remove-result):
* lisp/ob-fortran.el (org-babel-fortran-ensure-main-wrap):
* lisp/org-capture.el (org-capture-set-target-location):
* lisp/org-compat.el (org-maybe-keyword-time-regexp):
* lisp/org-table.el (org-table-expand-lhs-ranges):
Remove subsumed repetitions.
Remove subsumed repetitions in regexps
770f76f050376bbd77a3cc8cf44db57cf855a27c
Mattias Engdegård
Thu Feb 20 16:05:18 2020 +0100
* lisp/ob-core.el (org-babel-expand-noweb-references):
* lisp/ob-tangle.el (org-babel-tangle): Use `org-src-get-lang-mode' so
as to use proper major mode.
Reported-by: immanuel <immanuel.litzroth@gmail.com>
<http://lists.gnu.org/r/emacs-orgmode/2019-08/msg00272.html>
* lisp/ob-core.el (org-babel-execute-src-block): Fix behaviour of
":dir path" when ":mkdirp" is not defined.
* testing/lisp/test-ob.el (test-ob-core/dir-mkdirp): Add a test case.
TINYCHANGE
Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
> Could you provide a commit message for your patch?
* lisp/ob-core.el (org-babel-make-language-alias): Add edit-prep to
the list of defalias'ed functions.
* lisp/ob-core.el (org-babel-execute-src-block): Make directory if
":dir path" does not exist when ":mkdirp yes" exist.
* doc/org-manual.org (Header arguments): Document it.
* testing/lisp/test-ob.el: Add a specific testing file for ob-core.el,
and add a testing for ":mkdir yes" work with :dir header argument
usage.
* lisp/ob-core.el (org-babel--string-to-number): Fix the regular expression.
* testing/lisp/test-ob.el (test-ob/string-to-number): Test cases.
If people write the data in the form "0001"", it means that he wants to
treat it as a string.
TINYCHANGE
> #+name: TBL
> | id | name | age |
> |------|--------|-----|
> | 0001 | Apollo | 16 |
> | 0002 | Bmw | 16 |
>
> #+BEGIN_SRC emacs-lisp :results value pp :var tbl=TBL
> (mapc 'print tbl)
> #+END_SRC
>
> #+RESULTS:
> : (("0001" "Apollo" 16)
> : ("0002" "Bmw" 16))
TINYCHANGE
* lisp/ob-core.el (org-babel-strip-quotes): Remove function.
* lisp/org-macs.el (org-strip-quotes): New function.
* lisp/org-compat.el (org-babel-strip-quotes): Alias for new function.
* lisp/ob-haskell.el (org-babel-execute:haskell):
* lisp/ob-octave.el (org-babel-octave-evaluate-session): Use new function.
* lisp/ob-core.el (org-babel-execute-src-block): ":results file" must
be specified in order to return a file.
(org-babel-merge-params): :file and :file-ext no longer imply :results
file.
* testing/lisp/test-ob.el (test-ob/indented-cached-org-bracket-link):
(test-ob/result-file-link-type-header-argument):
(test-ob/result-graphics-link-type-header-argument): Update tests.
Deducing the results from some other arguments is not obvious.
Moreover, it prevents users from setting, e.g., :file-ext, in a node
property, as every block would then create a file.
Reported-by: Alex Fenton <alex@pressure.to>
<http://lists.gnu.org/r/emacs-orgmode/2018-05/msg00469.html>
* lisp/ob-core.el (org-babel-open-src-block-result): Fix docstring,
refactor code.
* lisp/org.el (org-open-at-point): Refactor so as to not call
`org-babel-open-src-block-result' unconditionally.
* lisp/ob-core.el (org-babel-sha1-hash): Add optional argument to
specify context.
(org-babel-execute-src-block): Use new argument.
* lisp/ob-exp.el (org-babel-exp-src-block): Use new argument.
(org-babel-exp-results): Fix context.
Reported-by: Ken Mankoff <mankoff@gmail.com>
<http://lists.gnu.org/r/emacs-orgmode/2018-06/msg00117.html>
* lisp/ob-core.el (org-babel-execute-src-block): Handle "link" :results
format.
* doc/org-manual.org: Add document for new result format "link".
* testing/lisp/test-ob.el (test-ob/result-file-link-type-header-argument):
New test.
