this-month-in-org/2022-02-30-orgnvim.txt

                     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                             FEBRUARY 2022
                      An orgmode clone for neovim

                            Kristijan Husak
                     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


                               2022-02-30


Timothy (TEC) here. This month we have a guest post from a different
part of the Org ecosystem, to highlight one of the most promising
efforts to provide a good experience outside Emacs.

//github.com/nvim-orgmode/orgmode

“But I use Emacs, I don’t care” you may say. In that case, I’d like to
point out that wider spread and better Org support enriches the Org
ecosystem as a whole.  It makes the format more approachable, and
/useful/ for other people. This is good for everybody.

Without any further ado, here’s the guest post kindly written by
Kristijan.  Enjoy!

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

Like every beginner Vim user, at some point I ran into a usual editor
war post: Vim vs Emacs. At that time, I didn’t have an idea what “Emacs”
was.

A simple Google search yielded something that seemed just like a very
simple editor with strange, but more familiar shortcuts. I didn’t bother
too much to figure out what it is, because I was already pulled in
fairly deep into Vim and its philosophy.


Note taking in (Neo)Vim
═══════════════════════

  At first, I did some note taking only when really necessary, in random
  plain text files. Most of the things I managed to keep in my head,
  since I was younger and less busy 🙂.

  Once I got into the situation where I needed to keep more notes,
  [vimwiki] was the natural choice.

  That worked very well for a period, until the need for writing quick
  notes arise. Vimwiki didn’t have anything that would allow that. I
  could of course have a mapping that opens a specific file where I can
  add notes, but that just never felt right in my mind. I would keep a
  bunch of things in the same place, and then later I needed to spend
  some time organizing them.

  At that point, I wasn’t sure how to achieve what I want. I did a brief
  look at [Emacs OrgMode] to see what’s all the fuss about, but to me,
  it seemed just like a different version of Markdown. You put some
  unordered lists as your notes, and that’s it. I never spent more time
  trying to see all the neat features. I even tried creating some of my
  custom note taking tools, but I never managed to finish them because I
  didn’t have a clear idea of how to solve my problems.


[vimwiki] <https://github.com/vimwiki/vimwiki>

[Emacs OrgMode] <https://orgmode.org/>


First encounter with Orgmode like tool: vim-dotoo
═════════════════════════════════════════════════

  One weekend, I was browsing through Vim subreddit, as I usually do at
  least once a day. There was a post about an “Orgmode like task
  logging” plugin called [vim-dotoo]. I opened it up, and I didn’t see
  much at that point. I wasn’t too excited. I went through readme, and
  noticed that author ([dhruvasagar]) put a fairly big emphasis on the
  “Agenda view”. I had no idea what “Agenda view” is.  Thankfully, the
  author also made a [screencast], which is rather long (1.5h), but I
  had some time, so I went through it.

  At that point, I was first met with “Capturing” and “Refiling”. *My
  mind was blown!* What a simple, yet extremely powerful idea! How had
  that never crossed my mind? From that point on, this plugin had my
  full attention.

  I’m always emphasizing that [dhruvasagar] and his [vim-dotoo] plugin
  are most deserving for having inspired
  <https://github.com/nvim-orgmode/orgmode>, and I can’t thank him
  enough for that.


[vim-dotoo] <https://github.com/dhruvasagar/vim-dotoo>

[dhruvasagar] <https://github.com/dhruvasagar>

[screencast] <https://www.youtube.com/watch?v=nsv33iOnH34>


First steps with vim-dotoo and birth of orgmode.nvim
════════════════════════════════════════════════════

  For some time, I was using [vim-dotoo]. I moved all of my Vimwiki
  notes to it. It was a breath of fresh air. Alongside that, I started
  getting more interest in the original Emacs Orgmode. I started
  noticing the differences, and some of the missing features that were
  now looking quite attractive. I made [few contributions] to
  vim-dotoo. As time passed, and my notes started to grow, things began
  being slow. I did some profiling, and figured out that it’s just a
  usual Vim problem, Vimscript performance. It was just too slow for
  certain things that Orgmode provides, and it would hardly get any
  better as more things are added.

  Separately from Vim and Vimscript, [Neovim] was on a stable `v0.4'
  release, and `v0.5' was still being developed. I was using Neovim from
  version 0.3, and was carefully following the progress on it. Lua was
  introduced as a first class citizen. A Bunch of new plugins arise from
  it. All the benchmarks showed that Lua outperforms Vimscript in almost
  everything. Besides the performance, Lua is a “normal” programming
  language, which means that support for it is much better.

  At that point, I became curious: Could Lua be the path to the faster
  Orgmode? I spent several days thinking about it. I wanted to give it a
  try. My biggest concern was that I had absolutely zero experience
  writing parsers. I had never written anything more complicated than an
  averagely complicated regex for purposes of parsing. I noticed that
  vim-dotoo also used regex to do the parsing, so that eased my mind a
  bit.

  One weekend, I started working on it. It was really interesting and
  challenging.  I spent a lot of my free time on it. At certain points,
  it seemed like hacking, since it was not a proper parsing. I tried to
  learn how to write a proper parser, but it was just too time consuming
  and complicated. I proceeded with the regex parsing to see how far I
  can go.

  Besides parsing, I had a few more challenges to overcome:


[vim-dotoo] <https://github.com/dhruvasagar/vim-dotoo>

[few contributions]
<https://github.com/dhruvasagar/vim-dotoo/pulls?q=is%3Apr+sort%3Aupdated-desc+author%3Akristijanhusak+is%3Aclosed>

[Neovim] <https://github.com/neovim/neovim>

Understanding the OrgMode syntax and all the functionality
──────────────────────────────────────────────────────────

  This is still the biggest challenge. I didn’t have any idea how big
  and robust OrgMode is. If I would know it at that time, I wouldn’t
  even jump on this train.  It’s really hard to grasp all of
  it. Considering I’ve only used it for around 8 months, I think I made
  some good progress on learning it.


Remote editing
──────────────

  By remote editing, I mean automatically updating content in the
  current or any other file. Few examples: adding/updating properties,
  managing tags, changing TODO states, archiving, refiling, remote
  editing from agenda view, etc.

  There is no built-in way to update content in another file through the
  Neovim API, without actually opening the file in an editor. I solved
  this by:

  • Saving as much position information as possible in the internal
    state, so I can pinpoint the correct location
  • Opening a file in a `1 row x 1 col' floating window and doing quick
    edits there


Working with dates
──────────────────

  From my experience, dates are challenging in all areas of programming,
  so this is not so surprising. There are some Lua plugins for dates,
  but those seemed a bit too basic for my use case, and I wanted to keep
  external plugins to the minimum. I went with a custom solution that
  uses Lua’s native dates, which has certain limitations, but works out
  for most of the things.


Highlighting, mostly in Agenda view
───────────────────────────────────

  Vim’s syntax engine is fairly old, but still very much used,
  especially in the Vim community. Implementation of tree-sitter
  slightly improved this experience in Neovim, because “Highlight
  matches” are found via tree-sitter, instead of a bunch of regexes.

  This helped me out later for the Org file itself, but agenda view is
  still something that’s built as a custom view. Old Syntax highlight
  engine would be really hard to add, because the content is too
  dynamic. I went with the Neovim highlight API that allows Highlighting
  things by their exact position in the buffer. Tree-sitter
  implementation does something similar in the background for
  Highlighting.


Keeping configuration simple and familiar to Emacs OrgMode
──────────────────────────────────────────────────────────

  Vim-dotoo configuration was mostly Vim style, through some global
  variables. I wanted to have a configuration that is familiar to an
  Emacs OrgMode user, by having as many options as possible named
  completely the same as in Emacs.

  For example, Here’s a comparison of few options between Emacs and
  Neovim:

  Emacs:

  ┌────
  │ (setq org-agenda-files '("~/orgmodes"))
  │ (setq org-agenda-skip-scheduled-if-done t)
  │ (setq org-agenda-span 7)
  │ (setq org-hide-leading-stars t)
  │  (setq org-capture-templates
  │        '(("t" "Todo" entry (file "~/orgmodes/todos.org")
  │       "* TODO %?")
  │      ("j" "Journal" entry (file "~/orgmodes/journal.org")
  │       "* %?\nEntered on %U\n   %a")))
  └────

  Neovim:

  ┌────
  │ require('orgmode').setup({
  │    org_agenda_files = { '~/orgmodes' },
  │    org_agenda_skip_scheduled_if_done = true,
  │    org_agenda_span = 7,
  │    org_hide_leading_stars = true
  │    org_capture_templates = {
  │       t = {
  │      description = 'Todo',
  │      target = '~/orgmodes/todos.org',
  │      template = '* TODO %?',
  │       },
  │       j = {
  │      description = 'Journal',
  │      target = '~/orgmodes/journal.org',
  │      template = '* %?\nEntered on %U\n   %a',
  │       }
  │    }
  │ })
  └────

  One of the most noticeable differences is between the usage of hyphens
  (`-') and underscores (`_'). I did that only for the sake of
  simplicity, because hyphens is not a valid character in variable names
  in Lua, so all of the options would need to be wrapped as a string
  (for example: `['org-agenda-files']').


First release of orgmode.nvim and introduction of tree-sitter parser
════════════════════════════════════════════════════════════════════

  After ~1.5 months I [published the initial version]. The focus was on
  Agenda and capturing (GTD), since those are the things I mostly
  used. It got some traction, and people started testing it and
  reporting bugs.

  One of the common questions was: /“Any plans to introduce tree-sitter
  parser?”/.

  I knew about [tree-sitter] and used it in my day-to-day job for a few
  programming languages, but I had absolutely no idea how it worked, and
  especially how to write a tree-sitter parser. I put it aside, and
  continued working on what I had.

  One day, Emilia ([milisims]) contacted me via email to ask me if I
  would be willing to try the tree-sitter parser she’s been working on
  for some time. I gladly accepted. She gave me access to the
  repository, and I started tinkering with it in a separate branch. No
  one was aware at that point that tree-sitter support would happen some
  time soon.

  After some time, I set up a “beta” branch called “tree-sitter” and
  [announced it for testing]. Once the reported bugs slowed to a
  trickle, I merged it into the “master” branch.

  I believe that tree-sitter grammar for Org could help out other
  editors to implement their version of Orgmode plugin, but I don’t
  think it would ever be helpful for Emacs. Emacs parser is the one and
  only that has it all implemented.  Also, as much as tree-sitter is
  powerful, its main purpose is to parse programming languages, which
  mostly has “static” patterns to match. Orgmode is by its nature
  dynamic, which causes a variety of issues for a parser that’s not
  meant for that kind of usage.


[published the initial version]
<https://www.reddit.com/r/neovim/comments/o8zp0k/orgmodenvim_orgmode_clone_written_in_lua_for/>

[tree-sitter] <https://github.com/tree-sitter/tree-sitter>

[milisims] <https://github.com/milisims>

[announced it for testing]
<https://www.reddit.com/r/neovim/comments/ph2xqc/orgmodenvim_treesitter_support/>


Limitations
═══════════

  (Neo)Vim is a great editor, but it still cannot compare to Emacs in
  certain things. Manipulating the “View” part of the editor is tricky
  or impossible for certain things.

  I even [made a label] for reported issues where Neovim support for
  certain things is a blocker. I’m hoping that at least some of these
  will be available in future Neovim releases.


[made a label]
<https://github.com/nvim-orgmode/orgmode/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Aneovim-dependency>


Features
════════

  I will not go into too many details about the available features,
  since those can be viewed in [repository readme], but I want to
  mention one feature that does not exist as a built/-in feature in the
  Emacs Orgmode: [Notifications].

  This allows getting a “desktop notification” for tasks that are within
  the specified threshold for schedule/deadline time. It requires some
  configuration to set up a cron job, but it’s been working great for me
  for several months now.


[repository readme]
<https://github.com/nvim-orgmode/orgmode#features-detailed-breakdown>

[Notifications]
<https://github.com/nvim-orgmode/orgmode/blob/master/DOCS.md#notifications-experimental>


Plans
═════

  The current state of the project is very usable for me. I’m not
  lacking any of the major features, mostly because I’m not used to
  using them. Nevertheless, there are plans to add more things, and I’m
  getting a lot of help from the community. I want to specifically
  mention [levouh] and [lukas-reineke], since they added a lot of value
  to the project, and I want to thank them and everyone else who
  contributed. Their help is much appreciated.

  There are few high priority tasks that I’m hoping to flush out first:

  • Implementing [v1.0.0] release of the tree-sitter parser. This should
    allow for faster and less error-prone parsing.
  • [Infrastructure for plugin developers], to allow other people to
    build plugins on top of nvim-orgmode.

  And a long term goal for these:

  • Tables support (and at least basic formulas)
  • [Org Babel like code block evaluation] (and hopefully basic support
    for literate programming)
  • [Diary format dates]
  • [Custom agenda commands]
  • More clocking features (reports)
  • File specific configuration via directives ([todo keywords],
    properties, etc.)


[levouh] <https://github.com/levouh>

[lukas-reineke] <https://github.com/lukas-reineke>

[v1.0.0] <https://github.com/milisims/tree-sitter-org/issues/13>

[Infrastructure for plugin developers]
<https://github.com/nvim-orgmode/orgmode/issues/26>

[Org Babel like code block evaluation]
<https://github.com/nvim-orgmode/orgmode/issues/190>

[Diary format dates]
<https://github.com/nvim-orgmode/orgmode/issues/195>

[Custom agenda commands]
<https://github.com/nvim-orgmode/orgmode/issues/135>

[todo keywords] <https://github.com/nvim-orgmode/orgmode/issues/185>


Closing thoughts
════════════════

  When I started working on [nvim-orgmode], I didn’t have a clue what
  I’m jumping into. Every day I learn about more and more Orgmode
  features that I wasn’t even aware existed.

  I’m certain that this project will never manage to clone the Orgmode
  functionality completely, but I’m hoping it will get close enough so
  everyone from Neovim community and Emacsers trying out Neovim will be
  able to use it for their needs.

  Having experienced Orgmode users [testing] it is a huge help, so if
  anyone is willing to give it a try, feel free to open up an issue and
  write your thoughts there. Thanks!


[nvim-orgmode] <https://github.com/nvim-orgmode/orgmode>

[testing] <https://github.com/nvim-orgmode/orgmode/issues/159>