(feature): use org-capture templates (#216)

Instead of implementing our own templating system, we abuse org-capture's templating system. We add 2 additional properties:

- :head: a starting template that goes at the beginning of the file.
- :file-name: a string that expands to the file name

The templates are customizable at `org-roam-capture-templates` and `org-roam-ref-capture-templates`.

CHANGELOG.md

@@ -1,19 +1,20 @@
 # Changelog
 
-## 0.1.3 (TBD)
+## 1.0.0
 
-The biggest change, by far, the shift of database storage into SQLite.
-This means that the org-roam cache needs to be built manually at least
-once via `M-x org-roam-build-cache`.
 ### Breaking Changes
 * [#200][gh-200] Move Org-roam cache into a SQLite database.
+* [#203][gh-203] Roam protocol is deprecated, in favour of extending org-roam-protocol.
 
 ### New Features
 * [#182][gh-182] Support file name aliases via `#+ROAM_ALIAS`.
-* [#188][gh-188] Add `org-roam-protocol`, shifting `roam://` link handling into Emacs-lisp.
 
 ### Features
-* [#165][gh-165] Add templating functionality via `org-roam-templates`.
+* [#216][gh-216] Adds templating functionality by extending org-capture.
+
+
+### Bugfixes
+* [#207][gh-207], [#221][gh-221] small bugfixes to Org-roam graph generation
 
 ## 0.1.2 (2020-02-21)
 
@@ -102,10 +103,12 @@ Mostly a documentation/cleanup release.
 [gh-141]: https://github.com/jethrokuan/org-roam/pull/141
 [gh-142]: https://github.com/jethrokuan/org-roam/pull/142
 [gh-143]: https://github.com/jethrokuan/org-roam/pull/143
-[gh-165]: https://github.com/jethrokuan/org-roam/pull/165
 [gh-182]: https://github.com/jethrokuan/org-roam/pull/182
 [gh-188]: https://github.com/jethrokuan/org-roam/pull/188
 [gh-200]: https://github.com/jethrokuan/org-roam/pull/200
+[gh-207]: https://github.com/jethrokuan/org-roam/pull/207
+[gh-216]: https://github.com/jethrokuan/org-roam/pull/216
+[gh-221]: https://github.com/jethrokuan/org-roam/pull/221
 
  # Local Variables:
  # eval: (auto-fill-mode -1)

doc/roam_protocol.md

@@ -1,29 +1,36 @@
 ## What is Roam protocol?
 
 Org-roam defines two protocols that help boost productivity, by
-extending `org-protocol`. 
+extending `org-protocol`: the `roam-file` and `roam-ref` protocol.
 
-The first protocol is the `roam-file` protocol. This is a simple
-protocol that opens the path specified by the `file` key (e.g.
-`org-protocol:/roam-file?file=/tmp/file.org`). This is used in the
-generated graph.
+## The `roam-file` protocol
 
-The second protocol is the `roam-ref` protocol. This protocol finds or
-creates a new note with a given `ROAM_KEY` (see
-[Anatomy](anatomy.md)).
+This is a simple protocol that opens the path specified by the `file`
+key (e.g. `org-protocol://roam-file?file=/tmp/file.org`). This is used
+in the generated graph.
+
+## The `roam-ref` Protocol
+
+This protocol finds or creates a new note with a given `ROAM_KEY` (see
+[Anatomy](anatomy.md)):
+
+![roam-ref](images/roam-ref.gif)
 
 To use this, create a Firefox bookmarklet as follows:
 
 ```javascript
 javascript:location.href =
-'org-protocol:/roam-ref?template=ref&ref='
+'org-protocol:/roam-ref?template=r&ref='
 + encodeURIComponent(location.href)
 + '&title='
 + encodeURIComponent(document.title)
 ```
 
-where `template` is the template you have defined for your web
-snippets. This template should contain a `#+ROAM_KEY: {ref}` in it.
+where `template` is the template key for a template in
+`org-roam-ref-capture-templates`. More documentation on the templating
+system can be found [here](templating.md).
+
+These templates should contain a `#+ROAM_KEY: {ref}` in it.
 
 ## Org-protocol Setup
 

doc/templating.md

@@ -0,0 +1,83 @@
+Rather than creating blank files on `org-roam-insert` and
+`org-roam-find-file`, it is may be desirable to prefill the file with
+content. This may include:
+
+- Time of creation
+- File it was created from
+- Clipboard content
+- Any other data you may want to input manually
+
+This requires a complex template insertion system, but fortunately,
+Org ships with a powerful one: `org-capture`. However, org-capture was
+not designed for such use. Org-roam abuses `org-capture` to some
+extent, extending its syntax. To first understand how org-roam's
+templating system works, it may be useful to look into org-capture.
+
+## Org-roam Templates
+
+The org-roam capture template extends org-capture's template with 2
+additional properties:
+
+1. `:file-name`: This is the file name template used when a new note
+   is created.
+2. `:head`: This is the template that is inserted on initial note
+   creation. 
+
+### Org-roam Template Expansion
+
+Org-roam's template definitions also extend org-capture's template
+syntax, to allow prefilling of strings. In many scenarios,
+`org-roam-capture` is passed a mapping between variables and strings.
+For example, during `org-roam-insert`, a title is prompted for. If the
+title doesn't already exist, we would like to create a new file,
+without prompting for the title again.
+
+Variables passed are expanded with the `${var}` syntax. For example,
+eduring `org-roam-insert`, `${title}` is prefilled for expansion. Any
+variables that do not contain strings, are prompted for values using
+`completing-read`.
+
+After doing this expansion, the org-capture's template expansion
+system is used to fill up the rest of the template. You may read up
+more on this on [org-capture's documentation
+page](https://orgmode.org/manual/Template-expansion.html#Template-expansion).
+
+For example, take the template: `"%<%Y%m%d%H%M%S>-${title}"`, with the title
+`"Foo"`. The template is first expanded into `%<%Y%m%d%H%M%S>-Foo`. Then
+org-capture expands `%<%Y%m%d%H%M%S>` with timestamp: e.g.
+`20200213032037-Foo`.
+
+This templating system is used throughout org-roam templates.
+
+### Template examples
+
+Here I walkthrough the default template, reproduced below.
+
+```
+("d" "default" plain (function org-roam--capture-get-point)
+     "%?"
+     :file-name "%<%Y%m%d%H%M%S>-${slug}"
+     :head "#+TITLE: ${title}\n"
+     :unnarrowed t)
+```
+
+1. The template has short key `"d"`. If you have only one template,
+   org-roam automatically chooses this template for you.
+2. The template is given a description of `"default"`.
+3. `plain` text is inserted. Other options include Org headings via
+   `entry`.
+4. `(function org-roam--capture-get-point)` should not be changed.
+5. `"%?"` is the template inserted on each call to `org-roam-capture`.
+   This template means don't insert any content, but place the cursor
+   here.
+6. `:file-name` is the file-name template for a new note, if it
+doesn't yet exist. This creates a file at path that looks like
+   `/path/to/org-roam-directory/20200213032037-foo.org`.
+7. `:head` contains the initial template to be inserted (once only),
+   at the beginning of the file. Here, the title global attribute is
+   inserted.
+8. `:unnarrowed t` tells org-capture to show the contents for the
+   whole file, rather than narrowing to just the entry.
+
+Other options you may want to learn about include `:immediate-finish`.
+

mkdocs.yml

@@ -9,6 +9,7 @@ nav:
 - Installation: installation.md
 - Configuration: configuration.md
 - Anatomy of a Roam file: anatomy.md
+- The Templating System: templating.md
 - Ecosystem: ecosystem.md
 - Similar Packages: comparison.md
 - "Appendix: Note-taking Workflow": notetaking_workflow.md

org-roam-protocol.el

@@ -31,6 +31,15 @@
 (require 'org-roam)
 
 (declare-function org-roam-find-ref "org-roam" (&optional info))
+(declare-function org-roam--capture-get-point "org-roam" ())
+
+(defvar org-roam-ref-capture-templates
+  '(("r" "ref" plain (function org-roam--capture-get-point)
+     ""
+     :file-name "${slug}"
+     :head "#+TITLE: ${title}
+#+ROAM_KEY: ${ref}\n"
+     :unnarrowed t)))
 
 (defun org-roam-protocol-open-ref (info)
   "Process an org-protocol://roam-ref?ref= style url with INFO.
@@ -38,10 +47,9 @@
 The sub-protocol used to reach this function is set in
 `org-protocol-protocol-alist'.
 
-This function decodes a ref, and places it into
-This function detects an file, and opens it.
+This function decodes a ref.
 
-  javascript:location.href = \\='org-protocol://roam-ref?ref=\\='+ \\
+  javascript:location.href = \\='org-protocol://roam-ref?template=r&ref=\\='+ \\
         encodeURIComponent(location.href) + \\='&title=\\=' \\
         encodeURIComponent(document.title) + \\='&body=\\=' + \\
         encodeURIComponent(window.getSelection())"
@@ -50,13 +58,21 @@ This function detects an file, and opens it.
                                        (let ((key (car k.v))
                                              (val (cdr k.v)))
                                          (cons key (org-link-decode val)))) alist)))
-    (when (assoc 'ref decoded-alist)
+    (unless (assoc 'ref decoded-alist)
+      (error "No ref key provided."))
+    (when-let ((title (cdr (assoc 'title decoded-alist))))
+      (push (cons 'slug (org-roam--title-to-slug title)) decoded-alist))
+    (let* ((org-roam-capture-templates org-roam-ref-capture-templates)
+           (org-roam--capture-context 'ref)
+           (org-roam--capture-info decoded-alist)
+           (template (cdr (assoc 'template decoded-alist))))
       (raise-frame)
-      (org-roam-find-ref decoded-alist)))
+      (org-roam-capture nil template)
+      (message "Item captured.")))
   nil)
 
 (defun org-roam-protocol-open-file (info)
-  "Process an org-protocol://roam-ref?ref= style url with INFO.
+  "This handler simply opens the file with emacsclient.
 
   Example protocol string:
 

org-roam.el

@@ -36,6 +36,7 @@
 ;;;; Library Requires
 (require 'org)
 (require 'org-element)
+(require 'org-capture)
 (require 'ob-core)  ;for org-babel-parse-header-arguments
 (require 'subr-x)
 (require 'dash)
@@ -69,12 +70,6 @@ All Org files, at any level of nesting, is considered part of the Org-roam."
   :type 'directory
   :group 'org-roam)
 
-(defcustom org-roam-new-file-directory nil
-  "Path to where new Org-roam files are created.
-If nil, default to the org-roam-directory (preferred)."
-  :type 'directory
-  :group 'org-roam)
-
 (defcustom org-roam-buffer-position 'right
   "Position of `org-roam' buffer.
 Valid values are
@@ -109,26 +104,6 @@ If nil, always ask for filename."
   :type 'boolean
   :group 'org-roam)
 
-(defcustom org-roam-graph-viewer (executable-find "firefox")
-  "Path to executable for viewing SVG."
-  :type 'string
-  :group 'org-roam)
-
-(defcustom org-roam-graphviz-executable (executable-find "dot")
-  "Path to graphviz executable."
-  :type 'string
-  :group 'org-roam)
-
-(defcustom org-roam-graph-max-title-length 100
-  "Maximum length of titles in Graphviz graph nodes."
-  :type 'number
-  :group 'org-roam)
-
-(defcustom org-roam-graph-node-shape "ellipse"
-  "Shape of Graphviz nodes."
-  :type 'string
-  :group 'org-roam)
-
 ;;;; Dynamic variables
 (defvar org-roam--current-buffer nil
   "Currently displayed file in `org-roam' buffer.")
@@ -596,21 +571,8 @@ specified via the #+ROAM_ALIAS property."
            (slug (-reduce-from #'replace title pairs)))
       (s-downcase slug))))
 
-;;;; New file creation
-(defvar org-roam-templates
-  (list (list "default" (list :file #'org-roam--file-name-timestamp-title
-                              :content "#+TITLE: ${title}")))
-  "Templates to insert for new files in org-roam.")
-
-(defun org-roam--file-name-timestamp-title (title)
-  "Return a file name (without extension) for new files.
-
-It uses TITLE and the current timestamp to form a unique title."
-  (let ((timestamp (format-time-string "%Y%m%d%H%M%S" (current-time)))
-        (slug (org-roam--title-to-slug title)))
-    (format "%s_%s" timestamp slug)))
-
-(defun org-roam--new-file-path (id &optional absolute)
+;;;; Org-roam capture
+(defun org-roam--new-file-path (id)
   "The file path for a new Org-roam file, with identifier ID.
 If ABSOLUTE, return an absolute file-path. Else, return a relative file-path."
   (let ((absolute-file-path (file-truename
@@ -618,44 +580,111 @@ If ABSOLUTE, return an absolute file-path. Else, return a relative file-path."
                               (if org-roam-encrypt-files
                                   (concat id ".org.gpg")
                                 (concat id ".org"))
-                              (or org-roam-new-file-directory
-                                  org-roam-directory)))))
-    (if absolute
-        absolute-file-path
-      (file-relative-name absolute-file-path
-                          (file-truename org-roam-directory)))))
-
-(defun org-roam--get-template (&optional template-key)
-  "Return an Org-roam template. TEMPLATE-KEY is used to get a template."
-  (unless org-roam-templates
-    (user-error "No templates defined"))
-  (if template-key
-      (cadr (assoc template-key org-roam-templates))
-    (if (= (length org-roam-templates) 1)
-        (cadar org-roam-templates)
-      (cadr (assoc (completing-read "Template: " org-roam-templates)
-                   org-roam-templates)))))
-
-(defun org-roam--make-new-file (&optional info)
-  (let ((template (org-roam--get-template (cdr (assoc 'template info))))
-        (title (or (cdr (assoc 'title info))
-                   (completing-read "Title: " nil)))
-        file-name-fn file-path)
-    (fset 'file-name-fn (plist-get template :file))
-    (setq file-path (org-roam--new-file-path (file-name-fn title) t))
-    (push (cons 'slug (org-roam--title-to-slug title)) info)
-    (unless (file-exists-p file-path)
-      (org-roam--touch-file file-path)
-      (write-region
-       (s-format (plist-get template :content)
-                 'aget
-                 info)
-       nil file-path nil))
-    (org-roam--db-update-file file-path)
+                              org-roam-directory))))
+    absolute-file-path))
+
+(defvar org-roam--capture-file-name-default "%<%Y%m%d%H%M%S>"
+  "The default file name format for org-roam templates.")
+
+(defvar org-roam--capture-header-default "#+TITLE: ${title}\n"
+  "The default file name format for org-roam templates.")
+
+(defvar org-roam--capture-file-path nil
+  "The file path for the Org-roam capture. This variable is set
+during the Org-roam capture process.")
+
+(defvar org-roam--capture-info nil
+  "An alist of additional information passed to the org-roam
+template. This variable is populated dynamically, and is only
+non-nil during the org-roam capture process.")
+
+(defvar org-roam--capture-context nil
+  "A cons cell containing the context (search term) to get the
+exact point in a file. This variable is populated dynamically,
+and is only active during an org-roam capture process.
+
+E.g. ('title . \"New Title\")")
+
+(defvar org-roam-capture-templates
+  '(("d" "default" plain (function org-roam--capture-get-point)
+     "%?"
+     :file-name "%<%Y%m%d%H%M%S>-${slug}"
+     :head "#+TITLE: ${title}\n"
+     :unnarrowed t))
+  "Capture templates for org-roam.")
+
+(defun org-roam--fill-template (str &optional info)
+  "Return a file name from template STR."
+  (-> str
+      (s-format (lambda (key)
+                  (or (s--aget info key)
+                      (completing-read (format "%s: " key ) nil))) nil)
+      (org-capture-fill-template)))
+
+(defun org-roam--capture-new-file ()
+  "Creates a new file, by reading the file-name attribute of the
+currently active org-roam template. Returns the path to the new file."
+  (let* ((name-templ (or (org-capture-get :file-name)
+                         org-roam--capture-file-name-default))
+         (new-id (s-trim (org-roam--fill-template
+                          name-templ
+                          org-roam--capture-info)))
+         (file-path (org-roam--new-file-path new-id)))
+    (org-roam--touch-file file-path)
+    (write-region
+     (org-roam--fill-template (or (org-capture-get :head)
+                                  org-roam--capture-header-default)
+                              org-roam--capture-info)
+     nil file-path nil)
+    (sleep-for 0.2)  ;; Hack: expand-file-name stringp nil error sporadically otherwise
+    file-path))
+
+(defun org-roam--capture-get-point ()
+  "Returns exact point to file for org-capture-template.
+The file to use is dependent on the context:
+
+If the search is via title, it is assumed that the file does not
+yet exist, and org-roam will attempt to create new file.
+
+If the search is via ref, it is matched against the Org-roam database.
+If there is no file with that ref, a file with that ref is created."
+  (pcase org-roam--capture-context
+    ('title
+     (let ((file-path (org-roam--capture-new-file)))
+       (setq org-roam--capture-file-path file-path)
+       (set-buffer (org-capture-target-buffer file-path))
+       (widen)
+       (goto-char (point-max))))
+    ('ref
+     (let* ((completions (org-roam--get-ref-path-completions))
+            (ref (cdr (assoc 'ref org-roam--capture-info)))
+            (file-path (or (cdr (assoc ref completions))
+                           (org-roam--capture-new-file))))
+       (setq org-roam--capture-file-path file-path)
+       (set-buffer (org-capture-target-buffer file-path))
+       (widen)
+       (goto-char (point-max))))
+    (_ (error "Invalid org-roam-capture-context."))))
+
+(defun org-roam-capture (&optional goto keys)
+  "Create a new file using an Org-roam template, and returns the
+path to the edited file. The templates are defined at
+`org-roam-capture-templates'."
+  (interactive "P")
+  (let ((org-capture-templates org-roam-capture-templates)
+        file-path)
+    (when (= (length org-capture-templates) 1)
+      (setq keys (caar org-capture-templates)))
+    (org-capture goto keys)
+    (setq file-path org-roam--capture-file-path)
+    (setq org-roam--capture-file-path nil)
     file-path))
 
 ;;; Interactive Commands
 ;;;; org-roam-insert
+(defvar org-roam--capture-insert-point nil
+  "The point to jump to after the call to `org-roam-insert'.")
+
 (defun org-roam-insert (prefix)
   "Find an org-roam file, and insert a relative org link to it at point.
 If PREFIX, downcase the title before insertion."
@@ -669,22 +698,44 @@ If PREFIX, downcase the title before insertion."
          (completions (org-roam--get-title-path-completions))
          (title (completing-read "File: " completions nil nil region-text))
          (region-or-title (or region-text title))
-         (absolute-file-path (or (cdr (assoc title completions))
-                                 (org-roam--make-new-file (list (cons 'title title)))))
+         (target-file-path (cdr (assoc title completions)))
          (current-file-path (-> (or (buffer-base-buffer)
                                     (current-buffer))
                                 (buffer-file-name)
                                 (file-truename)
-                                (file-name-directory))))
-    (when region ;; Remove previously selected text.
-      (goto-char (car region))
-      (delete-char (- (cdr region) (car region))))
-    (insert (format "[[%s][%s]]"
-                    (concat "file:" (file-relative-name absolute-file-path
-                                                        current-file-path))
-                    (format org-roam-link-title-format (if prefix
-                                                           (downcase region-or-title)
-                                                         region-or-title))))))
+                                (file-name-directory)))
+         (buf (current-buffer))
+         (p (point-marker)))
+    (unless (and target-file-path
+                 (file-exists-p target-file-path))
+      (let* ((org-roam--capture-info (list (cons 'title title)
+                                           (cons 'slug (org-roam--title-to-slug title))))
+             (org-roam--capture-context 'title))
+        (setq target-file-path (org-roam-capture))))
+    (with-current-buffer buf
+      (when region ;; Remove previously selected text.
+        (delete-region (car region) (cdr region)))
+      (let ((link-location (concat "file:" (file-relative-name target-file-path current-file-path)))
+            (description (format org-roam-link-title-format (if prefix
+                                                                (downcase region-or-title)
+                                                              region-or-title))))
+        (goto-char p)
+        (insert (format "[[%s][%s]]"
+                        link-location
+                        description))
+        (setq org-roam--capture-insert-point (point))))))
+
+(defun org-roam--capture-advance-point ()
+  "Advances the point if it is updated.
+We need this function because typically org-capture prevents the
+point from being advanced, whereas when a link is inserted, the
+point moves some characters forward. This is added as a hook to
+`org-capture-after-finalize-hook'."
+  (when org-roam--capture-insert-point
+    (goto-char org-roam--capture-insert-point)
+    (setq org-roam--capture-insert-point nil)))
+
+(add-hook 'org-capture-after-finalize-hook #'org-roam--capture-advance-point)
 
 ;;;; org-roam-find-file
 (defun org-roam--get-title-path-completions ()
@@ -705,10 +756,14 @@ If PREFIX, downcase the title before insertion."
   "Find and open an org-roam file."
   (interactive)
   (let* ((completions (org-roam--get-title-path-completions))
-         (title-or-slug (completing-read "File: " completions))
-         (absolute-file-path (or (cdr (assoc title-or-slug completions))
-                                 (org-roam--make-new-file (list (cons 'title title-or-slug))))))
-    (find-file absolute-file-path)))
+         (title (completing-read "File: " completions))
+         (file-path (cdr (assoc title completions))))
+    (if file-path
+        (find-file file-path)
+      (let* ((org-roam--capture-info (list (cons 'title title)
+                                           (cons 'slug (org-roam--title-to-slug title))))
+             (org-roam--capture-context 'title))
+        (org-roam-capture '(4))))))
 
 ;;;; org-roam-find-ref
 (defun org-roam--get-ref-path-completions ()
@@ -724,11 +779,8 @@ INFO is an alist containing additional information."
   (interactive)
   (let* ((completions (org-roam--get-ref-path-completions))
          (ref (or (cdr (assoc 'ref info))
-                  (completing-read "Ref: " (org-roam--get-ref-path-completions))))
-         (file-path (cdr (assoc ref completions))))
-    (if file-path
-        (find-file file-path)
-      (find-file (org-roam--make-new-file info)))))
+                  (completing-read "Ref: " (org-roam--get-ref-path-completions) nil t))))
+    (find-file (cdr (assoc ref completions)))))
 
 ;;;; org-roam-switch-to-buffer
 (defun org-roam--get-roam-buffers ()
@@ -756,10 +808,15 @@ INFO is an alist containing additional information."
 ;;;; Daily notes
 (defun org-roam--file-for-time (time)
   "Create and find file for TIME."
-  (let* ((org-roam-templates (list (list "daily" (list :file (lambda (title) title)
-                                                       :content "#+TITLE: ${title}")))))
-    (org-roam--make-new-file (list (cons 'title (format-time-string "%Y-%m-%d" time))
-                                   (cons 'template "daily")))))
+  (let* ((title (format-time-string "%Y-%m-%d" time))
+         (org-roam-capture-templates (list (list "d" "daily" 'plain (list 'function #'org-roam--capture-get-point)
+                                                 ""
+                                                 :immediate-finish t
+                                                 :file-name "${title}"
+                                                 :head "#+TITLE: ${title}")))
+         (org-roam--capture-context 'title)
+         (org-roam--capture-info (list (cons 'title title))))
+    (org-roam-capture)))
 
 (defun org-roam-today ()
   "Create and find file for today."
@@ -786,7 +843,6 @@ INFO is an alist containing additional information."
     (let ((path (org-roam--file-for-time time)))
       (org-roam--find-file path))))
 
-
 ;;; The org-roam buffer
 ;;;; org-roam-link-face
 (defface org-roam-link
@@ -968,6 +1024,28 @@ Valid states are 'visible, 'exists and 'none."
     ('none (org-roam--setup-buffer))))
 
 ;;; The graphviz links graph
+;;;; Options
+(defcustom org-roam-graph-viewer (executable-find "firefox")
+  "Path to executable for viewing SVG."
+  :type 'string
+  :group 'org-roam)
+
+(defcustom org-roam-graphviz-executable (executable-find "dot")
+  "Path to graphviz executable."
+  :type 'string
+  :group 'org-roam)
+
+(defcustom org-roam-graph-max-title-length 100
+  "Maximum length of titles in Graphviz graph nodes."
+  :type 'number
+  :group 'org-roam)
+
+(defcustom org-roam-graph-node-shape "ellipse"
+  "Shape of Graphviz nodes."
+  :type 'string
+  :group 'org-roam)
+
+;;;; Functions
 (defun org-roam--build-graph ()
   "Build the Graphviz string.
 The Org-roam database titles table is read, to obtain the list of titles.