org-mode/lisp
Ihor Radchenko fc80d052db
Re-implement org-element-cache and add headline support
* lisp/org-element.el (org-element-with-disabled-cache): New macro.

(org-element-greater-elements): Add new org-data element.  It
functions like a virtual headline containing the whole buffer.  The
org-data properties are like headlie properties, but according to the
top-level drawer.  org-data's category is the buffer's category as
defined by top-level property drawer, #+CATEGORY keyworsd, and the
buffer file name.

(org-element--cache-element-properties, org-element-set-element): New
variable containing properties to be transferred when updating changed
element in cache in `org-element-set-element'.

(org-element--get-node-properties): Allow parsing node propreties in
top-level drawer when new optional argument is passed.  Respect
PROPERTY+ syntax.

(org-element--get-global-node-properties): New function.  It returns
node properties for top-level property drawer.

(org-element-org-data-parser, org-element-org-data-interpreter):
Implement the new org-data element.

(org-element-headline-parser, org-element-section-parser): Add new
:robust-begin and :robust-end
properties delimiting safe changes that do not modify headline
element.

(org-element--list-struct): Fix cache update when adding a headline
inside list.

(org-element--current-element): Implement cache support.  Record
parsing mode (:mode) and parsing granularity (:granularity) in the
element properties.

(org-element-parse-buffer, org-element--next-mode): Support new
org-data element.

(org-element--parse-elements): Record parsing granularity in the
returned tree

(org-element-use-cache): Enable cache by default.

(org-element-cache-persistent): New variable controlling cache
persistance across sessions.  Enabled by default.

(org-element--cache-self-verify,
org-element--cache-self-verify-frequency,
org-element--cache-diagnostics, org-element--cache-map-statistics,
org-element--cache-map-statistics-threshold,
org-element--cache-diagnostics-level,
org-element--cache-diagnostics-ring,
org-element--cache-diagnostics-ring-size): New variables controlling
cache diagnostics and self-diagnostics.  Greatly simplifies cache
debugging.

(org-element--cache, org-element--cache-sync-requests,
org-element--cache-sync-timer): Make cache buffer-local by default.

(org-element--headline-cache): Implement separate cache storing only
headlines and inlinetasks.

(org-element--cache-size, org-element--headline-cache-size): New
variables containing cache sizes.  This is much faster than
`avl-tree-size'.

(org-element--cache-sync-requests): Update docstring explaning the
request list structure.

(org-element--cache-sync-keys-value): New variable replacing
`org-element--cache-sync-keys' hash table.  The hash table was not
reliable because it was using elements as keys.  Upon any cached
element update/shift, the keys were invalidated making cache ordering
incorrect and breaking the cache badly.  Now, the cache keys are
stored as :org-element--cache-sync-key element property and the new
variable stores marker value indicating the current sync request
cycle.  See `org-element--cache-key' for more details.

(org-element--cache-change-tic): New variable controlling buffer
modification count that is registered in cache.  This variable allows
catching "stealth" edits.

(org-element--cache-non-modifying-commands): New variable listing
commands that will not be slown down if we fill cache on the fly.

(org-element--request-key, org-element--request-beg,
org-element--request-end, org-element--request-offset,
org-element--request-parent, org-element--request-phase): New macros.
They improve code readability (especially when using nameless-mode).

(org-element--format-element, org-element--cache-log-message,
org-element--cache-warn): New macros implementing generic logging
functionality.

(org-element--cache-key): Add section and org-data element support.
Change cache key storage from hash map to :org-element--cache-sync-key
element property + `org-element--cache-sync-keys-value'.  We use the
latter to group all the cache keys during a single cache request
sequence.  Once sync request is fully complete, the
`org-element--cache-sync-keys-value' is updated making all the old
sync keys obsolete (they will still be store as element properties).

(org-element--headline-cache-root): New function returning headline
cache root.

(org-element--cache-active-p): Prevent cache updates when
`inhibit-modification-hooks' is non-nil, unless non-nil optional
argument is provided.

(org-element--cache-find): Share cache between indirect buffers and
the base buffer.  We have to do it because after-change hooks for
indirect buffer are not called in the base buffer and vice versa.  Add
support for section and org-data elements.

(org-element--cache-put): Implement new approach for cache key
storage.  Add diagnostics.  Indicate cached elements using :cached
element property.  Support cache size calculation.

(org-element--cache-remove): Invalidate parent contents when removing
element.  Support cache size calculation.  Detect cache corruption due
to misordered elements.

(org-element--cache-shift-positions): Support :robust-begin and
:robust-end element properties.

(org-element--cache-sync): Add diagnostics.  Add detailed comments.
Prevent slowdown when large cache chunks need to be deleted forcing
O(N) complexity cutoff.  In phase 2, fix cases when next request
contains deleted cache key.  In phase 2, fix scenario when newly
inserted element intersects with existing elements in cache.  In phase
2, detect obsolete parents removed from cache.

(org-element--open-end-p): New function checking if an element can
have blank lines right after its :contents-end.

(org-element--parse-to): Do not alter match data.  Process complex
parsing mode changes correctly.  Support headlines in cache.  Support
org-data parsing.  Add detailed comments.  Add diagnostics.

(org-element--cache-sensitive-re): Make list lines sensitive.

(org-element--cache-change-warning): Update docstring.  Now, the
variable can have t, nil, and number values.  Numbers are used to
provide more details about changed headlines.

(org-element--cache-before-change, org-element--cache-after-change):
Handle headline hierarchy.  Properly handle cache in indirect
buffers.

(org-element--cache-after-change): Update docstring clarifying the
return values.  Add special handling for headline and org-data
elements updating them in-place instead of removing together with the
whole contents when possible.  Use :robust-begin/:robust-end element
properties to detect robust changes.

(org-element--cache-submit-request): Add detailed comments.  Correctly
handle cache in indirect buffers.  Delegate element modifications to
`org-element--cache-for-removal'.

(org-element--cache-verify-element): New function for cache
self-verification.

(org-element--cache-persist-before-write,
org-element--cache-persist-before-read,
org-element--cache-persist-after-read): Implement cache persistance.

(org-element-cache-reset): Correctly handle cache in indirect
buffers.  Support cache persistance.  Support new cache size
calculation and new cache key schema.

(org-element-cache-map): New function analagous to `org-element-map',
but much faster.  The function overperforms org-ql written by Adam
Porter aka alphapapa [1] and reuses some ideas from there (namely,
fast element skipping via regexps).

[1] https://github.com/alphapapa/org-ql/

(org-element-at-point): The returned elements are now guaranteed to
have correct parents up to org-data.  New optional argument
CACHED-ONLY limits element search to current cache---if element is not
in cache and current command is not in cache
`org-element--cache-non-modifying-commands', the cache is not updated
and the function returns nil.  Also, support cache verification.

(org-element-at-point-no-context): New function.  It is analogous of
older `org-element-at-point' with no guarantee that :parent properties
are correct beyond direct parent heading.  This function does not
update cache and can be useful when cache updates should be avoided
for performance reasons.

* lisp/ob-core.el (org-babel-where-is-src-block-result): Support
section and org-data elements in cache.

* lisp/org-macro.el (org-macro-replace-all,
org-macro--find-keyword-value): Support org-element-cache.

* lisp/org-table.el (orgtbl-to-generic): Support org-element-cache.

* lisp/org.el (org-mode): Add cache persistance.

(org-up-element): Preserve old behaviour when error is returned for
section and org-data element.

*
testing/lisp/test-org-archive.el (test-org-archive/update-status-cookie):
Fix test when cache is active.

* testing/lisp/test-org-colview.el (test-org-colview/columns-update):
Fix test.

* testing/lisp/test-org-element.el (test-org-element/extract-element):
Add suport for new org-data element.

* testing/lisp/test-org-element.el (test-org-element/parent-property):
Fix equality check.  Parents returned by cache and `org-element-map'
may not be `eq' now.  Just `equal'.

* testing/lisp/test-org-element.el (test-org-element/context): Support
section and headline parents.
2021-10-16 23:22:18 +08:00
..
Makefile Move link-related core functions out of "org.el" 2019-03-10 18:00:27 +01:00
ob-C.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ob-R.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ob-awk.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-calc.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-clojure.el Backport commit 8b07994e2 from Emacs 2021-03-25 00:56:53 -04:00
ob-comint.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-core.el Re-implement org-element-cache and add headline support 2021-10-16 23:22:18 +08:00
ob-css.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-ditaa.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-dot.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ob-emacs-lisp.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-eshell.el Fix 108e5c7bd 2021-05-30 18:56:05 +02:00
ob-eval.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ob-exp.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-forth.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-fortran.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ob-gnuplot.el ob-gnuplot: Honour :missing argument 2021-10-01 11:44:52 +08:00
ob-groovy.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ob-haskell.el Fix org-babel-header-args:haskell not being an alist 2021-09-21 16:51:54 +08:00
ob-java.el Backport commit 2075411ed from Emacs 2021-09-30 21:18:08 -04:00
ob-js.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ob-julia.el Merge branch 'km/from-emacs-28' into bugfix 2021-10-02 00:51:56 -04:00
ob-latex.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-lilypond.el Merge branch 'bugfix' 2021-09-29 13:19:55 +02:00
ob-lisp.el Prefer HTTPS to HTTP in most links 2021-03-21 15:21:22 -04:00
ob-lob.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-lua.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ob-makefile.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ob-matlab.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-maxima.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-ocaml.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-octave.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-org.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-perl.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ob-plantuml.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-processing.el Prefer HTTPS to HTTP in most links 2021-03-21 15:21:22 -04:00
ob-python.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-ref.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-ruby.el Prefer HTTPS to HTTP in most links 2021-03-21 15:21:22 -04:00
ob-sass.el Prefer HTTPS to HTTP in most links 2021-03-21 15:21:22 -04:00
ob-scheme.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ob-screen.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ob-sed.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-shell.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ob-sql.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ob-sqlite.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ob-table.el Backport commit c78e16962 from Emacs 2021-09-26 22:49:55 -04:00
ob-tangle.el lisp/ob-tangle.el: Add variable declaration 2021-09-26 14:34:41 +02:00
ob.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
oc-basic.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
oc-biblatex.el Merge branch 'bugfix' 2021-10-10 23:11:35 +02:00
oc-bibtex.el oc-bibtex: Pacify byte-compiler 2021-10-08 22:45:22 +02:00
oc-csl.el oc-csl: Map § and §§ o section locators 2021-10-11 23:58:17 +02:00
oc-natbib.el Backport commit 4341e79a5 from Emacs 2021-10-02 14:20:38 -04:00
oc.el oc: Add autoload cookie for `org-cite-insert' 2021-10-13 22:22:22 +02:00
ol-bbdb.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ol-bibtex.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ol-docview.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ol-doi.el Backport commit d50597189 from Emacs 2021-10-01 21:57:10 -04:00
ol-eshell.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ol-eww.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
ol-gnus.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ol-info.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
ol-irc.el Backport commit 348b2aed0 from Emacs 2021-07-11 22:40:57 -04:00
ol-man.el lisp/ol-man.el: Add copyright header 2021-10-08 15:38:01 +02:00
ol-mhe.el Update copyright year to 2021 2021-01-01 14:59:01 -05:00
ol-rmail.el Update Carsten's email address in some files 2021-05-07 16:50:57 +02:00
ol-w3m.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ol.el Backport commit 62d6cecfc from Emacs 2021-10-02 13:56:50 -04:00
org-agenda.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
org-archive.el Update Carsten's email address in some files 2021-05-07 16:50:57 +02:00
org-attach-git.el lisp/org-attach-git.el: New option `org-attach-git-dir' 2021-05-15 16:13:49 +02:00
org-attach.el org-attach.el: Fix checkdoc warnings 2021-10-02 19:15:52 +02:00
org-capture.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
org-clock.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
org-colview.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
org-compat.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
org-crypt.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
org-ctags.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
org-datetree.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
org-duration.el duration: Avoid recursive load error 2021-01-04 00:30:00 -05:00
org-element.el Re-implement org-element-cache and add headline support 2021-10-16 23:22:18 +08:00
org-entities.el Update Carsten's email address in some files 2021-05-07 16:50:57 +02:00
org-faces.el Add faces to improve contextuality of agenda views 2021-09-26 10:44:14 +02:00
org-feed.el Update Carsten's email address in some files 2021-05-07 16:50:57 +02:00
org-footnote.el footnote: Improve last commit 2021-10-06 14:41:50 +02:00
org-goto.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
org-habit.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
org-id.el org-id: Fix checkdoc warnings 2021-09-29 08:50:06 +02:00
org-indent.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
org-inlinetask.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
org-install.el Use lexical-binding cookie in some trivial files 2021-01-24 22:36:30 -05:00
org-keys.el Backport commit 4341e79a5 from Emacs 2021-10-02 14:20:38 -04:00
org-lint.el org-lint: Do not require space after : in keywords 2021-10-04 14:14:16 +08:00
org-list.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
org-macro.el Re-implement org-element-cache and add headline support 2021-10-16 23:22:18 +08:00
org-macs.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
org-mobile.el Update Carsten's email address in some files 2021-05-07 16:50:57 +02:00
org-mouse.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
org-num.el Backport commit 62d6cecfc from Emacs 2021-10-02 13:56:50 -04:00
org-pcomplete.el Various minor docfixes found by checkdoc 2021-09-16 21:24:59 +02:00
org-persist.el lisp/org-persist.el: New library to store data across sessions 2021-10-16 21:16:11 +08:00
org-plot.el Backport commit f1f57a541 from Emacs 2021-10-01 21:57:09 -04:00
org-protocol.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
org-refile.el Merge branch 'maint' 2021-05-25 02:13:55 +02:00
org-src.el org-src.el: Fix checkdoc warnings 2021-10-02 17:10:15 +02:00
org-table.el Re-implement org-element-cache and add headline support 2021-10-16 23:22:18 +08:00
org-tempo.el Backport commit a0854f939 from Emacs 2021-03-14 17:57:07 -04:00
org-timer.el Prune Emacs 25.1 compatibility kludges 2021-10-04 23:43:14 -04:00
org.el Re-implement org-element-cache and add headline support 2021-10-16 23:22:18 +08:00
ox-ascii.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ox-beamer.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ox-html.el Backport commit f9111d878 from Emacs 2021-10-02 13:44:34 -04:00
ox-icalendar.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ox-koma-letter.el Backport commit bd60fca2f from Emacs 2021-10-02 11:02:13 -04:00
ox-latex.el Backport commit f9111d878 from Emacs 2021-10-02 13:44:34 -04:00
ox-man.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ox-md.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ox-odt.el *el: Fix indentation 2021-09-29 09:22:47 +02:00
ox-org.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ox-publish.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ox-texinfo.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00
ox.el Fix email addresses in some *.el headers 2021-10-03 08:54:33 +02:00