From 2040b613914cb4153efee3d11650777a4c759d33 Mon Sep 17 00:00:00 2001 From: Nicolas Goaziou Date: Tue, 13 Jun 2017 10:41:13 +0200 Subject: [PATCH 1/2] org-element: Remove outdated comments --- lisp/org-element.el | 78 ++++++--------------------------------------- 1 file changed, 10 insertions(+), 68 deletions(-) diff --git a/lisp/org-element.el b/lisp/org-element.el index f4b2a39e1..847e7989d 100644 --- a/lisp/org-element.el +++ b/lisp/org-element.el @@ -22,79 +22,21 @@ ;;; Commentary: ;; -;; Org syntax can be divided into three categories: "Greater -;; elements", "Elements" and "Objects". +;; See for details about +;; Org syntax. ;; -;; Elements are related to the structure of the document. Indeed, all -;; elements are a cover for the document: each position within belongs -;; to at least one element. -;; -;; An element always starts and ends at the beginning of a line. With -;; a few exceptions (`clock', `headline', `inlinetask', `item', -;; `planning', `property-drawer', `node-property', `section' and -;; `table-row' types), it can also accept a fixed set of keywords as -;; attributes. Those are called "affiliated keywords" to distinguish -;; them from other keywords, which are full-fledged elements. Almost -;; all affiliated keywords are referenced in -;; `org-element-affiliated-keywords'; the others are export attributes -;; and start with "ATTR_" prefix. -;; -;; Element containing other elements (and only elements) are called -;; greater elements. Concerned types are: `center-block', `drawer', -;; `dynamic-block', `footnote-definition', `headline', `inlinetask', -;; `item', `plain-list', `property-drawer', `quote-block', `section' -;; and `special-block'. -;; -;; Other element types are: `babel-call', `clock', `comment', -;; `comment-block', `diary-sexp', `example-block', `export-block', -;; `fixed-width', `horizontal-rule', `keyword', `latex-environment', -;; `node-property', `paragraph', `planning', `src-block', `table', -;; `table-row' and `verse-block'. Among them, `paragraph' and -;; `verse-block' types can contain Org objects and plain text. -;; -;; Objects are related to document's contents. Some of them are -;; recursive. Associated types are of the following: `bold', `code', -;; `entity', `export-snippet', `footnote-reference', -;; `inline-babel-call', `inline-src-block', `italic', -;; `latex-fragment', `line-break', `link', `macro', `radio-target', -;; `statistics-cookie', `strike-through', `subscript', `superscript', -;; `table-cell', `target', `timestamp', `underline' and `verbatim'. -;; -;; Some elements also have special properties whose value can hold -;; objects themselves (e.g. an item tag or a headline name). Such -;; values are called "secondary strings". Any object belongs to -;; either an element or a secondary string. -;; -;; Notwithstanding affiliated keywords, each greater element, element -;; and object has a fixed set of properties attached to it. Among -;; them, four are shared by all types: `:begin' and `:end', which -;; refer to the beginning and ending buffer positions of the -;; considered element or object, `:post-blank', which holds the number -;; of blank lines, or white spaces, at its end and `:parent' which -;; refers to the element or object containing it. Greater elements, -;; elements and objects containing objects will also have -;; `:contents-begin' and `:contents-end' properties to delimit -;; contents. Eventually, All elements have a `:post-affiliated' -;; property referring to the buffer position after all affiliated -;; keywords, if any, or to their beginning position otherwise. -;; -;; At the lowest level, a `:parent' property is also attached to any -;; string, as a text property. -;; -;; Lisp-wise, an element or an object can be represented as a list. +;; Lisp-wise, a syntax object can be represented as a list. ;; It follows the pattern (TYPE PROPERTIES CONTENTS), where: -;; TYPE is a symbol describing the Org element or object. +;; TYPE is a symbol describing the object. ;; PROPERTIES is the property list attached to it. See docstring of -;; appropriate parsing function to get an exhaustive -;; list. -;; CONTENTS is a list of elements, objects or raw strings contained -;; in the current element or object, when applicable. +;; appropriate parsing function to get an exhaustive list. +;; CONTENTS is a list of syntax objects or raw strings contained +;; in the current object, when applicable. ;; -;; An Org buffer is a nested list of such elements and objects, whose -;; type is `org-data' and properties is nil. +;; For the whole document, TYPE is `org-data' and PROPERTIES is nil. ;; -;; The first part of this file defines Org syntax, while the second -;; one provide accessors and setters functions. +;; The first part of this file defines constants for the Org syntax, +;; while the second one provide accessors and setters functions. ;; ;; The next part implements a parser and an interpreter for each ;; element and object type in Org syntax. From 1e92f5ed39541a473f295b414e8c89f6b0390f83 Mon Sep 17 00:00:00 2001 From: Kaushal Modi Date: Tue, 13 Jun 2017 11:41:14 -0400 Subject: [PATCH 2/2] Allow org-file-contents to fetch file contents from a URL * lisp/org.el (org--file-cache): New variable. (org-reset-file-cache): (org-file-url-p): New function. (org-mode-restart): Use new function. * lisp/org.el (org-file-contents): Allow the FILE argument to be a URL. If the URL contents are already cached, return the cache contents, else download the file and return contents of that. The file is automatically cached each time it is downloaded. Add a new optional argument NOCACHE. If this is non-nil, the URL is always downloaded afresh. Use `org--file-cache' and `org-file-url-p'. * lisp/org.el (org-edit-special): Do not allow editing the "file" if a URL is specified for the "#+SETUPFILE". * lisp/ox.el (org-export--list-bound-variables) (org-export--prepare-file-contents): * lisp/org-macro.el (org-macro--collect-macros) : Adapt to the possibility that the input to `org-file-contents' can be a URL too. * doc/org.texi (Export settings, In-buffer settings) (The very busy C-c C-c key): Mention that #+SETUPFILE keyword can now take a URL as a value, and that C-c C-c on the #+SETUPFILE line will clear the org file cache. * testing/lisp/test-org.el (test-org/org-file-contents-url) (test-org/org-file-contents-file): Add tests for org-file-contents. * testing/lisp/test-ox.el (test-org-export/get-inbuffer-options): Add test for reading setupfile specified via a URL. --- doc/org.texi | 38 +++++++++++--------- etc/ORG-NEWS | 12 ++++++- lisp/org-macro.el | 22 +++++++----- lisp/org.el | 74 ++++++++++++++++++++++++++++++++------- lisp/ox.el | 38 ++++++++++++-------- testing/lisp/test-org.el | 75 ++++++++++++++++++++++++++++++++++++++++ testing/lisp/test-ox.el | 32 +++++++++++++++++ 7 files changed, 238 insertions(+), 53 deletions(-) diff --git a/doc/org.texi b/doc/org.texi index 83016282c..36d6aa7aa 100644 --- a/doc/org.texi +++ b/doc/org.texi @@ -10406,14 +10406,14 @@ override options set at a more general level. @cindex #+SETUPFILE In-buffer settings may appear anywhere in the file, either directly or -indirectly through a file included using @samp{#+SETUPFILE: filename} syntax. -Option keyword sets tailored to a particular back-end can be inserted from -the export dispatcher (@pxref{The export dispatcher}) using the @code{Insert -template} command by pressing @key{#}. To insert keywords individually, -a good way to make sure the keyword is correct is to type @code{#+} and then -to use @kbd{M-@key{TAB}}@footnote{Many desktops intercept @kbd{M-TAB} to -switch windows. Use @kbd{C-M-i} or @kbd{@key{ESC} @key{TAB}} instead.} for -completion. +indirectly through a file included using @samp{#+SETUPFILE: filename or URL} +syntax. Option keyword sets tailored to a particular back-end can be +inserted from the export dispatcher (@pxref{The export dispatcher}) using the +@code{Insert template} command by pressing @key{#}. To insert keywords +individually, a good way to make sure the keyword is correct is to type +@code{#+} and then to use @kbd{M-@key{TAB}}@footnote{Many desktops intercept +@kbd{M-TAB} to switch windows. Use @kbd{C-M-i} or @kbd{@key{ESC} @key{TAB}} +instead.} for completion. The export keywords available for every back-end, and their equivalent global variables, include: @@ -17174,14 +17174,16 @@ have a lower ASCII number than the lowest priority. This line sets a default inheritance value for entries in the current buffer, most useful for specifying the allowed values of a property. @cindex #+SETUPFILE -@item #+SETUPFILE: file -The setup file is for additional in-buffer settings. Org loads this file and -parses it for any settings in it only when Org opens the main file. @kbd{C-c -C-c} on the settings line will also parse and load. Org also parses and -loads the file during normal exporting process. Org parses the contents of -this file as if it was included in the buffer. It can be another Org file. -To visit the file, @kbd{C-c '} while the cursor is on the line with the file -name. +@item #+SETUPFILE: file or URL +The setup file or a URL pointing to such file is for additional in-buffer +settings. Org loads this file and parses it for any settings in it only when +Org opens the main file. If URL is specified, the contents are downloaded +and stored in a temporary file cache. @kbd{C-c C-c} on the settings line +will parse and load the file, and also reset the temporary file cache. Org +also parses and loads the document during normal exporting process. Org +parses the contents of this document as if it was included in the buffer. It +can be another Org file. To visit the file (not a URL), @kbd{C-c '} while +the cursor is on the line with the file name. @item #+STARTUP: @cindex #+STARTUP Startup options Org uses when first visiting a file. @@ -17422,7 +17424,9 @@ If any highlights shown in the buffer from the creation of a sparse tree, or from clock display, remove such highlights. @item If the cursor is in one of the special @code{#+KEYWORD} lines, scan the -buffer for these lines and update the information. +buffer for these lines and update the information. Also reset the Org file +cache used to temporary store the contents of URLs used as values for +keywords like @code{#+SETUPFILE}. @item If the cursor is inside a table, realign the table. The table realigns even if automatic table editor is turned off. diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index eb0e11c4a..83972d4e9 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -203,7 +203,7 @@ manual for details. **** Add global macros through ~org-export-global-macros~ With this variable, one can define macros available for all documents. **** New keyword ~#+EXPORT_FILE_NAME~ -Simiralry to ~:EXPORT_FILE_NAME:~ property, this keyword allows the +Similarly to ~:EXPORT_FILE_NAME:~ property, this keyword allows the user to specify the name of the output file upon exporting the document. This also has an effect on publishing. **** Horizontal rules are no longer ignored in LaTeX table math mode @@ -240,6 +240,16 @@ which causes refile targets to be prefixed with the buffer’s name. This is particularly useful when used in conjunction with ~uniquify.el~. +*** ~org-file-contents~ now allows the FILE argument to be a URL. +This allows ~#+SETUPFILE:~ to accept a URL instead of a local file +path. The URL contents are auto-downloaded and saved to a temporary +cache ~org--file-cache~. A new optional argument ~NOCACHE~ is added +to ~org-file-contents~. + +*** ~org-mode-restart~ now resets the newly added ~org--file-cache~. +Using ~C-c C-c~ on any keyword (like ~#+SETUPFILE~) will reset the +that file cache. + ** Removed functions *** Org Timeline diff --git a/lisp/org-macro.el b/lisp/org-macro.el index 6758d31f0..828c5e9e3 100644 --- a/lisp/org-macro.el +++ b/lisp/org-macro.el @@ -55,7 +55,8 @@ (declare-function org-element-macro-parser "org-element" ()) (declare-function org-element-property "org-element" (property element)) (declare-function org-element-type "org-element" (element)) -(declare-function org-file-contents "org" (file &optional noerror)) +(declare-function org-file-contents "org" (file &optional noerror nocache)) +(declare-function org-file-url-p "org" (file)) (declare-function org-in-commented-heading-p "org" (&optional no-inheritance)) (declare-function org-mode "org" ()) (declare-function vc-backend "vc-hooks" (f)) @@ -102,16 +103,21 @@ Return an alist containing all macro templates found." (if old-cell (setcdr old-cell template) (push (cons name template) templates)))) ;; Enter setup file. - (let ((file (expand-file-name - (org-unbracket-string "\"" "\"" val)))) - (unless (member file files) + (let* ((uri (org-unbracket-string "\"" "\"" (org-trim val))) + (uri-is-url (org-file-url-p uri)) + (uri (if uri-is-url + uri + (expand-file-name uri)))) + ;; Avoid circular dependencies. + (unless (member uri files) (with-temp-buffer - (setq default-directory - (file-name-directory file)) + (unless uri-is-url + (setq default-directory + (file-name-directory uri))) (org-mode) - (insert (org-file-contents file 'noerror)) + (insert (org-file-contents uri 'noerror)) (setq templates - (funcall collect-macros (cons file files) + (funcall collect-macros (cons uri files) templates))))))))))) templates)))) (funcall collect-macros nil nil))) diff --git a/lisp/org.el b/lisp/org.el index 2101ec7d1..191990db7 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -181,6 +181,8 @@ Stars are put in group 1 and the trimmed body in group 2.") (declare-function org-export-get-environment "ox" (&optional backend subtreep ext-plist)) (declare-function org-latex-make-preamble "ox-latex" (info &optional template snippet?)) +(defvar ffap-url-regexp) ;Silence byte-compiler + (defsubst org-uniquify (list) "Non-destructively remove duplicate elements from LIST." (let ((res (copy-sequence list))) (delete-dups res))) @@ -5280,17 +5282,62 @@ a string, summarizing TAGS, as a list of strings." (setq current-group (list tag)))) (_ nil))))) -(defun org-file-contents (file &optional noerror) - "Return the contents of FILE, as a string." - (if (and file (file-readable-p file)) +(defvar org--file-cache (make-hash-table :test #'equal) + "Hash table to store contents of files referenced via a URL. +This is the cache of file URLs read using `org-file-contents'.") + +(defun org-reset-file-cache () + "Reset the cache of files downloaded by `org-file-contents'." + (clrhash org--file-cache)) + +(defun org-file-url-p (file) + "Non-nil if FILE is a URL." + (require 'ffap) + (string-match-p ffap-url-regexp file)) + +(defun org-file-contents (file &optional noerror nocache) + "Return the contents of FILE, as a string. + +FILE can be a file name or URL. + +If FILE is a URL, download the contents. If the URL contents are +already cached in the `org--file-cache' hash table, the download step +is skipped. + +If NOERROR is non-nil, ignore the error when unable to read the FILE +from file or URL. + +If NOCACHE is non-nil, do a fresh fetch of FILE even if cached version +is available. This option applies only if FILE is a URL." + (let* ((is-url (org-file-url-p file)) + (cache (and is-url + (not nocache) + (gethash file org--file-cache)))) + (cond + (cache) + (is-url + (with-current-buffer (url-retrieve-synchronously file) + (goto-char (point-min)) + ;; Move point to after the url-retrieve header. + (search-forward "\n\n" nil :move) + ;; Search for the success code only in the url-retrieve header. + (if (save-excursion (re-search-backward "HTTP.*\\s-+200\\s-OK" nil :noerror)) + ;; Update the cache `org--file-cache' and return contents. + (puthash file + (buffer-substring-no-properties (point) (point-max)) + org--file-cache) + (funcall (if noerror #'message #'user-error) + "Unable to fetch file from %S" + file)))) + (t (with-temp-buffer - (insert-file-contents file) - (buffer-string)) - (funcall (if noerror 'message 'error) - "Cannot read file \"%s\"%s" - file - (let ((from (buffer-file-name (buffer-base-buffer)))) - (if from (concat " (referenced in file \"" from "\")") ""))))) + (condition-case err + (progn + (insert-file-contents file) + (buffer-string)) + (file-error + (funcall (if noerror #'message #'user-error) + (error-message-string err))))))))) (defun org-extract-log-state-settings (x) "Extract the log state setting from a TODO keyword string. @@ -20687,7 +20734,9 @@ Otherwise, return a user error." (format "[[%s]]" (expand-file-name (let ((value (org-element-property :value element))) - (cond ((not (org-string-nw-p value)) + (cond ((org-file-url-p value) + (user-error "The file is specified as a URL, cannot be edited")) + ((not (org-string-nw-p value)) (user-error "No file to edit")) ((string-match "\\`\"\\(.*?\\)\"" value) (match-string 1 value)) @@ -20951,7 +21000,8 @@ Use `\\[org-edit-special]' to edit table.el tables")) (funcall major-mode) (hack-local-variables) (when (and indent-status (not (bound-and-true-p org-indent-mode))) - (org-indent-mode -1))) + (org-indent-mode -1)) + (org-reset-file-cache)) (message "%s restarted" major-mode)) (defun org-kill-note-or-show-branches () diff --git a/lisp/ox.el b/lisp/ox.el index 53d35bba8..3b793a00f 100644 --- a/lisp/ox.el +++ b/lisp/ox.el @@ -1499,17 +1499,20 @@ Assume buffer is in Org mode. Narrowing, if any, is ignored." (cond ;; Options in `org-export-special-keywords'. ((equal key "SETUPFILE") - (let ((file - (expand-file-name - (org-unbracket-string "\"" "\"" (org-trim val))))) + (let* ((uri (org-unbracket-string "\"" "\"" (org-trim val))) + (uri-is-url (org-file-url-p uri)) + (uri (if uri-is-url + uri + (expand-file-name uri)))) ;; Avoid circular dependencies. - (unless (member file files) + (unless (member uri files) (with-temp-buffer - (setq default-directory - (file-name-directory file)) - (insert (org-file-contents file 'noerror)) + (unless uri-is-url + (setq default-directory + (file-name-directory uri))) + (insert (org-file-contents uri 'noerror)) (let ((org-inhibit-startup t)) (org-mode)) - (funcall get-options (cons file files)))))) + (funcall get-options (cons uri files)))))) ((equal key "OPTIONS") (setq plist (org-combine-plists @@ -1647,17 +1650,22 @@ an alist where associations are (VARIABLE-NAME VALUE)." "BIND") (push (read (format "(%s)" val)) alist) ;; Enter setup file. - (let ((file (expand-file-name - (org-unbracket-string "\"" "\"" val)))) - (unless (member file files) + (let* ((uri (org-unbracket-string "\"" "\"" val)) + (uri-is-url (org-file-url-p uri)) + (uri (if uri-is-url + uri + (expand-file-name uri)))) + ;; Avoid circular dependencies. + (unless (member uri files) (with-temp-buffer - (setq default-directory - (file-name-directory file)) + (unless uri-is-url + (setq default-directory + (file-name-directory uri))) (let ((org-inhibit-startup t)) (org-mode)) - (insert (org-file-contents file 'noerror)) + (insert (org-file-contents uri 'noerror)) (setq alist (funcall collect-bind - (cons file files) + (cons uri files) alist)))))))))) alist))))) ;; Return value in appropriate order of appearance. diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el index e55ee077b..35674baf4 100644 --- a/testing/lisp/test-org.el +++ b/testing/lisp/test-org.el @@ -6498,6 +6498,81 @@ Paragraph" (org-show-set-visibility 'minimal) (org-invisible-p2)))) +(ert-deftest test-org/org-file-contents-file () + "Test `org-file-contents' with a file as input." + (should + (string= "#+BIND: variable value +#+DESCRIPTION: l2 +#+LANGUAGE: en +#+SELECT_TAGS: b +#+TITLE: b +#+PROPERTY: a 1 +" (org-file-contents (expand-file-name "setupfile3.org" + (concat org-test-dir "examples/"))))) + + (let ((invalid-file "this-file-must-not-exist")) + ;; Throw error when trying to access an invalid file. + (should-error + (org-file-contents invalid-file)) + ;; Try to access an invalid file, but do not throw an error. + (should + (string-match-p "\\`Opening input file: No such file or directory" + (org-file-contents invalid-file :noerror))))) + +(ert-deftest test-org/org-file-contents-url () + "Test `org-file-contents' with a URL as input." + (should + (string= "foo" + (let ((buffer (generate-new-buffer "url-retrieve-output"))) + (unwind-protect + ;; Simulate successful retrieval of a URL. + (cl-letf (((symbol-function 'url-retrieve-synchronously) + (lambda (&rest_) + (with-current-buffer buffer + (insert "HTTP/1.1 200 OK\n\nfoo")) + buffer))) + (org-file-contents "http://some-valid-url")) + (kill-buffer buffer))))) + + (let ((invalid-url "http://this-url-must-not-exist")) + ;; Throw error when trying to access an invalid URL. + (should-error + (let ((buffer (generate-new-buffer "url-retrieve-output"))) + (unwind-protect + ;; Simulate unsuccessful retrieval of a URL. + (cl-letf (((symbol-function 'url-retrieve-synchronously) + (lambda (&rest_) + (with-current-buffer buffer + (insert "HTTP/1.1 404 Not found\n\ndoes not matter")) + buffer))) + (org-file-contents invalid-url)) + (kill-buffer buffer)))) + ;; Try to access an invalid URL, but do not throw an error. + (should-error + (let ((buffer (generate-new-buffer "url-retrieve-output"))) + (unwind-protect + ;; Simulate unsuccessful retrieval of a URL. + (cl-letf (((symbol-function 'url-retrieve-synchronously) + (lambda (&rest_) + (with-current-buffer buffer + (insert "HTTP/1.1 404 Not found\n\ndoes not matter")) + buffer))) + (org-file-contents invalid-url)) + (kill-buffer buffer)))) + (should + (string= + (format "Unable to fetch file from \"%s\"" invalid-url) + (let ((buffer (generate-new-buffer "url-retrieve-output"))) + (unwind-protect + ;; Simulate unsuccessful retrieval of a URL. + (cl-letf (((symbol-function 'url-retrieve-synchronously) + (lambda (&rest_) + (with-current-buffer buffer + (insert "HTTP/1.1 404 Not found\n\ndoes not matter")) + buffer))) + (org-file-contents invalid-url :noerror)) + (kill-buffer buffer))))))) + (provide 'test-org) diff --git a/testing/lisp/test-ox.el b/testing/lisp/test-ox.el index 69a778bbb..72b6c8ccd 100644 --- a/testing/lisp/test-ox.el +++ b/testing/lisp/test-ox.el @@ -232,6 +232,38 @@ num:2 <:active"))) org-test-dir) (org-export--get-inbuffer-options)) '(:language "fr" :select-tags ("a" "b" "c") :title ("a b c")))) + ;; Options set through SETUPFILE specified using a URL. + (let ((buffer (generate-new-buffer "url-retrieve-output"))) + (unwind-protect + ;; Simulate successful retrieval of a setupfile from URL. + (cl-letf (((symbol-function 'url-retrieve-synchronously) + (lambda (&rest_) + (with-current-buffer buffer + (insert "HTTP/1.1 200 OK + +# Contents of http://link-to-my-setupfile.org +#+BIND: variable value +#+DESCRIPTION: l2 +#+LANGUAGE: en +#+SELECT_TAGS: b +#+TITLE: b +#+PROPERTY: a 1 +")) + buffer))) + (should + (equal + (org-test-with-temp-text + "#+DESCRIPTION: l1 +#+LANGUAGE: es +#+SELECT_TAGS: a +#+TITLE: a +#+SETUPFILE: \"http://link-to-my-setupfile.org\" +#+LANGUAGE: fr +#+SELECT_TAGS: c +#+TITLE: c" + (org-export--get-inbuffer-options)) + '(:language "fr" :select-tags ("a" "b" "c") :title ("a b c"))))) + (kill-buffer buffer))) ;; More than one property can refer to the same buffer keyword. (should (equal '(:k2 "value" :k1 "value")