I’m working to port some clj to cljs and needed a cljs version of line-seq. Having spent most of my cljs time in the browser, using NodeJS is a bit of a new world. There’s almost certainly a package to read files line by line, but I decided to roll my own because:

  • It’d be good experience reading through NodeJS docs
  • I’d have to wrap some cljs anyway
  • I’m a programmer :)

I saw this post on streams and decided to convert it to cljs.

First of, we load in our NodeJS libs:

(def fs (js/require "fs"))
(def stream (js/require "stream"))
(def ^:const eol (.-EOL (js/require "os"))) ;;eg "\n" or "\r\n"

As we read the file stream, we want to transform it from generic chunks of data to lines of text. We define a transform fn that we’ll use on our stream:

(defn- transform [chunk encoding done]
  (this-as this
    (let [data (if (.-_lastLineData this)
                 (str (.-_lastLineData this) chunk)
                 (str chunk))
          lines (clojure.string/split data (js/RegExp. eol "g"))]
      (set! (.-_lastLineData this) (last lines))
      (doseq [line (butlast lines)]
        (.push this line))
      (done))))

We also want to make sure we flush the last bit of data we read:

(defn- flush [done]
  (this-as this
    (if (.-_lastLineData this)
      (.push this (.-_lastLineData this)))
    (set! (.-_lastLineData this) nil)
    (done)))

Lastly, we set up our file stream and pipe it through our transformer:

(defn read-file-cb [file-name cb]
  (let [line-reader (.Transform stream #js {:objectMode true})
        source (.createReadStream fs file-name)]
    (set! (.-_transform line-reader) transform)
    (set! (.-_flush line-reader) flush)
    (.pipe source line-reader)
    (.on line-reader "readable"
         (fn []
           (when-let [line (.read line-reader)]
             (cb (str line)) ;;callback with each line
             (recur))))
    nil))

I generally prefer using channels instead of passing around callbacks, so I actually used this version with core.async:

(defn read-file-chan [file-name out-chan]
  (let [line-reader (.Transform stream #js {:objectMode true})
        source (.createReadStream fs file-name)]
    (set! (.-_transform line-reader) transform)
    (set! (.-_flush line-reader) flush)
    (.pipe source line-reader)
    (.on line-reader "readable"
         (fn []
           (go
             (loop []
               (when-let [line (.read line-reader)]
                 (>! out-chan (str line)) ;;push line to chan
                 (recur))))))
    nil))

This worked, but I really want the laziness of line-seq without the asynchronous code. In my next post, I’ll implement line-seq to work just the way we want.

Here’s a gist of the full code.

Find this post useful? Share the love!

Recent Posts