~iany/ Posts

Power of Monoid, Beauty of Simplicity

me@iany.me (Ian Yang) — Fri, 20 Feb 2026 00:00:00 +0800

A monoid is one of the smallest useful abstractions in algebra: a set closed under an associative binary operation, with an identity element. That simplicity is exactly why it shows up everywhere—from summing numbers and concatenating strings to powering divide-and-conquer algorithms and elegant data structures like finger trees. This post walks through what monoids are, why they give you “compute power” for free when you can phrase a problem in terms of them, and how to think about choosing the right monoid and predicate when you do.

What is a monoid?

A monoid is a set $S$ equipped with a binary operator $\bullet$ and an identity element $e$ (Wikipedia).

The operator is closed on $S$ . For all $a, b \in S$ , the result $a \bullet b$ is also in $S$ .
The operator is associative: For all $a,b,c \in S$ , $(a \bullet b) \bullet c = a \bullet (b \bullet c)$
The identity element $e$ satisfies $e \bullet a = a \bullet e = a$ for all $a \in S$ .

For example, The integer numbers with the operator addition (+) is a monoid, where the identity element is 0. The integer numbers with the operator multiplication (x) is also a monoid with the identity element 1.

The set of finite lists with the operator concatenation is a monoid since:

The operator is closed because concatenation of two finite lists is also a finite list.
The operator is associative, because both $(a \bullet b) \bullet c$ and $a \bullet (b \bullet c)$ result in a new list by placing elements of $a, b, c$ consecutively.
The identity element is the empty list.

The integer numbers with the operator max is a counterexample. The operator is closed and associative, but there is no identity element. Given any integer $e$ , there’s always a smaller integer $a$ such that $e \bullet a = e \ne a$ . However, max on the integer set with a lower bound is a monoid, such as the non-negative integers where the identity element is the lower bound 0.

Divide and conquer: why associativity matters

At first glance, associativity may seem too trivial to be useful in programming. However, associativity is what enables powerful divide-and-conquer strategies, where problems can be split into parts, solved independently, and then safely recombined.

Exponentiation by Squaring

Let’s begin with a simple application: repeatedly applying the binary operator to the same element.

 $\underbrace{a \bullet a \bullet \cdots \bullet a}_{a \text{ appears } n \text{ times}}$

Instead of applying the binary operator $n-1$ times sequentially, we exploit associativity to group every two instances of $a$ together recursively. This gives us a smaller problem when n is even:

 $\underbrace{(a \bullet a) \bullet \cdots \bullet (a \bullet a)}_{(a \bullet a) \text{ appears } \frac{n}{2} \text{ times}}$

When n is odd:

 $a \bullet \underbrace{(a \bullet a) \bullet \cdots \bullet (a \bullet a)}_{(a \bullet a) \text{ appears } \frac{n-1}{2} \text{ times}}$

We only need to compute $a \bullet a$ once to turn the problem of size $n$ to size $n/2$ . Repeating the process on $(a \bullet a)$ gives the Exponentiation by Squaring algorithm, which requires at most $\displaystyle 2 \lfloor \log _{2}n\rfloor$ computations that is more efficient than $n-1$ when $n$ is greater than 4.

For integers or real numbers under multiplication (×), exponentiation by squaring is an efficient algorithm to compute positive integer powers. Since the set of elliptic curve points under point addition forms a monoid, this same method can also be used to compute Elliptic Curve Scalar Multiplication.

General Divide-and-Conquer Search Algorithm

We can generalize the divide-and-conquer method to search an element in a sequence based solely on associativity.

The result of applying the monoid operator to a sequence from left to right serves as a summary of that sequence. If a predicate can determine whether a given element is in the sequence based solely on this summary, we can devise a general divide-and-conquer search algorithm.

Let’s say we want to search for a target element in the sequence $t_1, \ldots, t_n$ , where each $t_i$ belongs to a monoid $(S, \bullet, e)$ . Here, $S$ is the underlying set, $\bullet$ is the binary operation, and $e$ is the identity element.

We don’t know which kind of predicates works for the search algorithm. Let’s give a best guess that the predicate $p$ is a function of the monoid “summary” that $p(t_1 \bullet \cdots \bullet t_n)$ is true if and only if $t$ is in the sequence $t_1, \dots, t_n$ .

Assume that $p(t_1 \bullet \cdots \bullet t_n)$ is true, thus the target element $t$ is in the sequence. We divide the sequence into two halves: $t_1,\ldots,t_k$ and $t_{k+1},\ldots,t_n$ , where $1 \le k \le n$ . We then evaluate $p(t_1 \bullet \cdots \bullet t_k)$ to determine whether the target element lies in the first or second half, and continue the search.

Based on this observation, we can deduce the following property of the predicate: there exists an index $x$ such that

 $p(t_1 \bullet \cdots \bullet t_k) := \begin{cases} \text{false} & \text{if } k < x, \\ \text{true} & \text{if } k \ge x. \end{cases}$

$t_x$ is the target element if such $x$ exists; otherwise, the target element does not exist in the sequence.

Intuitively, the target element is the turning point at which the predicate on the running summary changes from false to true.

Note that $p$ makes sense only on the summary of any prefix of the sequence. If we need to continue the search in the second half, we must remember the summary of the scanned prefix.

Now we can define the search algorithm $\mathrm{Search}(p, s, \{t_i,\ldots,t_j\})$ where

$s$ is the summary of scanned prefix $t_1 \bullet \cdots \bullet t_{i-1}$ when $i > 1$ or the identity element $e$ otherwise.
$t_i, \ldots, t_j$ is the sub-range to search next.
$p$ is the predicate as defined above

The algorithm proceeds as follows:

If $p(s \bullet t_i \bullet \cdots \bullet t_j$ ) is false, the target element does not exist. The algorithm aborts with an error.
Otherwise, if there’s only one element ( $i = j$ ), $t_i$ is the target element. The algorithm aborts with the found result.
Otherwise, choose a pivot index $i \le m \lt j$ to split the sequence into two nonempty halves: $t_i, \ldots, t_m$ and $t_{m+1},\ldots,t_j$ . Test $p(s \bullet t_i \bullet \cdots \bullet t_m)$ that
- If it is true, continue the search in the first half: $\mathrm{Search}(p, s, \{t_i,\ldots,t_m\})$
- Otherwise, continue the search in the second half: $\mathrm{Search}(p, s \bullet (t_i \bullet \cdots \bullet t_m),\{t_{m+1},\ldots,t_j\})$

The algorithm starts with $\mathrm{Search}(p, e, \{t_1, \ldots, t_k\})$ .

Application: Random-Access Sequence

An application of the search algorithm is accessing the nth element in the sequence.

We initialize the sequence to all 1s and use the monoid of non-negative integers with addition $(\mathbb{N},+,0)$ :

 $\underbrace{1, \ldots, 1}_{n \text{ times}}$

The predicate to find the i-th (starting from 0) element is:

 $\[ p_i(s) := s > i \]$

It may seem silly to search for the i-th 1 in a sequence of 1s, but we can store any data in the sequence and attach the monoid values as annotations to guide the search algorithm.

Application: Max-Priority Queue

Another application is finding the element with the max priority.

We use the monoid of non-negative integers with operator max $(\mathbb{N},\mathrm{max},0)$ and assume that the maximum value has the maximum priority.

The predicate to find the element with the max priority is

 $\[ p(s) := s = m \]$

Where $m$ is the monoid summary of the entire sequence—that is, the maximum value in the sequence. The predicate checks whether the summary equals to $m$ .

Annotated Search Tree

A natural way to support the divide-and-conquer search is an annotated binary tree. Store the sequence elements at the leaves, and at each node store the monoid summary of the subtree—e.g. the sum of lengths or the maximum priority in that subtree. The predicate can then be evaluated on the left subtree’s annotation to decide whether to descend left or right, and the prefix summary is updated when going right by combining it with the left subtree’s summary.

A plain binary tree can degenerate to a list in the worst case, so operations may become linear. A more advanced structure, the finger tree¹, keeps the tree balanced and supports efficient access at both ends and in the middle; each node carries a monoidal “measure” of its subtree, and the same search strategy applies. In Haskell, Data.Sequence from the containers library implements sequences as finger trees with size (length) as the measure, giving $O(\log n)$ indexing, splitting, and concatenation.

Utility of the Identity Element

The general divide-and-conquer algorithm does not require a monoid—only a semigroup. A semigroup is a fancy word for a set equipped with a closed, associative binary operator but lacking an identity element. The presence of an identity element makes monoids convenient to work with.

The identity element serves as a natural default value or starting point for algorithms. For instance, in the search algorithm, the summary of the scanned prefix is initialized to $e$ . Without an identity element, we need an additional flag to indicate whether any prefix has been scanned, and the algorithm would have to branch conditionally based on that flag.

The art of choosing monoid and predicate

In the random-access example we used $(\mathbb{N}, +, 0)$ and annotated each position with $1$ —the summary of a segment is its length, and the predicate $s > i$ tells us whether the $i$ -th element lies in the prefix we have so far. In the max-priority queue we used $(\mathbb{N}, \max, 0)$ (or a bounded variant): the summary is the maximum value in the segment, and the predicate $s = m$ identifies the segment that contains the global maximum. In both cases, the monoid was chosen so that the combined summary over a range is exactly what the predicate needs to decide where to go next.

The flip side is that finding both the right monoid and the right predicate can be tricky. At each step the search has access only to the monoid summary of the prefix (or segment) seen so far, so the predicate must be decided from that summary alone. The monoid must be rich enough to supply the information the predicate needs. Sometimes the natural summary (e.g. sum or max) suggests the predicate (e.g. $s > i$ or $s = m$ ). Sometimes you must try a different carrier or operation, or encode extra information into the monoid (e.g. pairs or custom types), so that the predicate can be expressed. There is no universal recipe—it is a matter of design and experimentation. Reframe the problem as: “What do I need to know about a segment to decide the next step?” Then choose a monoid that can represent that knowledge and a predicate that uses it.

Hinze, R., & Paterson, R. (2006). Finger trees: A simple general-purpose data structure. Journal of Functional Programming, 16(2), 197–217. Cambridge University Press. https://www.cs.ox.ac.uk/ralf.hinze/publications/FingerTrees.pdf

Use Bun for Shell Scripts

me@iany.me (Ian Yang) — Sat, 07 Feb 2026 00:00:00 +0800

I’ve moved most of my small scripts to Bun. It sidesteps Windows’ lack of shebang support and the Git-for-Windows/WSL bash dance, and gives you one runtime for both ad-hoc shell-style commands and full scripts.

Problems Bun solves

Windows doesn’t execute #!/usr/bin/env python3 or #!/bin/bash, so scripts written for Linux or macOS usually need a separate PowerShell version. Bun compiles JavaScript to native executables, so a single script can run cross-platform without shebangs.

Mise file tasks have started to support shebangs on Windows in the latest release, but they rely on system bash.exe, which often points at WSL. Bun does not. You run bun ./script.js and embed shell commands via Bun.$; the script uses Bun’s bundled bash environment.

Why Bun for shell scripts

The command bun build ./script.ts --compile --outfile script produces a single binary, so no shebang or interpreter is needed on Windows. The Bun.build API is available too; I use a build.js script to walk a directory and compile all executables. Pre-built binaries start quickly—Bun’s docs cite ~5ms vs Node’s ~25ms for a simple script, which helps for small, frequently run scripts.

Bun has strong support for shell-style scripting. Bun Shell runs bash commands; the main entry point is Bun.$, which constructs shell commands from template literals for safe interpolation, similar to google/zx. On Windows, Bun ships with MSYS, so you get a consistent bash environment and common Linux CLI tools without extra setup.

Other useful features:

Bun.secrets—Securely store and retrieve secrets using the OS keystore (Windows Credential Manager, macOS Keychain, Linux libsecret). Secrets - Bun
Built-in support for glob patterns, YAML parsing, ANSI colors, and more. Bun Runtime

Short example

import { $ } from "bun";
const out = await $`echo hello`.text();
console.log(out); // hello

Run it with bun run script.js or bun ./script.js.

Obsidian Notes

JavaScript Shell Scripting

Use tmux for PowerShell in Windows Terminal

me@iany.me (Ian Yang) — Thu, 29 Jan 2026 23:30:42 +0800

You can get tmux session persistence and multiplexing in Windows Terminal by running tmux inside WSL and setting the default shell to PowerShell. New panes and windows will then start pwsh.exe instead of a Linux shell. Here’s a minimal setup using small wrappers and the tmux -C attach trick to configure new sessions.

Why tmux under WSL with PowerShell?

Since tmux does not run natively on Windows, the standard approach is to use it through WSL. However, if you launch wsl tmux from Windows Terminal, all panes will default to your WSL shell. When working in a Windows directory, I prefer to use native PowerShell within the Windows environment.

To ensure every tmux pane runs PowerShell, configure tmux to use pwsh.exe as its default command. When you launch pwsh.exe from WSL within a Windows directory, it brings us back to the Windows environment.

Wrappers so you can call tmux from PowerShell

Put these in a directory on your PATH (e.g. ~/Documents/PowerShell/bin).

PowerShell — tmux.ps1:

wsl tmux $args

Cmd — tmux.cmd:

@echo off
wsl tmux %*

From PowerShell or Cmd you can then run tmux new, tmux attach, etc., and they all go to WSL’s tmux.

Creating PowerShell Sessions

The next piece is a script that creates a new session whose default command is pwsh.exe, or attaches if that session already exists. Example usage: tmux-pwsh dev or tmux-pwsh work.

The script sets the default command only for sessions it creates; it does not affect sessions started with a standard tmux new-session.

tmux-pwsh.ps1:

param(
[Parameter(Mandatory=$true)]
[string]$SessionName
)
$pwshCommand = "exec pwsh.exe -nologo"
# Check if session exists using exact name match
wsl tmux has-session -t "\=$SessionName" 2>$null
if ($LASTEXITCODE -ne 0) {
wsl tmux new-session -d -s $SessionName $pwshCommand
echo "set-option default-command `"$pwshCommand`"" | `
wsl tmux -C attach -t "\=$SessionName" >$null
}
wsl tmux attach -t "\=$SessionName"

Note:

The = prefix does an exact session name match and must be escaped when sending the command from PowerShell to WSL.
The new session specifies command pwsh.exe via the command argument.
Running wsl tmux set-option immediately after creating the session does not work because there’s a brief window where the session is not available for setting options. Instead, sending set-option via tmux -C attach reliably applies the setting.

After this, splitting panes and creating new windows in that session will all start pwsh.exe -nologo in Windows Terminal.

Summary

Use WSL for tmux and PowerShell as the default command by setting tmux’s default command to exec pwsh.exe -nologo.
tmux.ps1 / tmux.cmd — thin wrappers so tmux from Windows means wsl tmux.
tmux-pwsh.ps1 <name> — create (or attach to) a named session that uses PowerShell as the default command.

Once these are on your PATH, run e.g. tmux-pwsh dev from Windows Terminal to get a tmux session where every pane is PowerShell.

Backup Ignored Files with Git Remote Branch

me@iany.me (Ian Yang) — Fri, 12 Dec 2025 03:22:58 +0800

When working with Git repositories, there are often files that need to be backed up but shouldn’t be committed to the main branch. These might include local development settings, IDE configuration files, personal notes, or development scripts that are specific to your workflow. The challenge is finding a way to back up these ignored files without polluting the main repository history.

The Original Solution

My original approach was to use a separate repository to store backup files for all repositories and create symbolic links. This worked, but had several drawbacks:

Backup files were disconnected from their source repositories, making it harder to track what belongs where
Backing up new files required moving them to the backup repository and then creating symbolic links

A Better Approach: Remote Branch

Instead of using a separate repository, we can use a remote branch within the same repository. This approach offers several advantages:

Everything stays in one repository
When you clone the repository, the backup branch comes with it
Backup files are stored in a separate branch, keeping the main branch clean
Full Git history for backup files, just like any other branch

The Script

With AI assistance, I developed git-store-file, a tool for managing ignored files by storing them in a remote branch (default: origin/_store). This keeps backup files separate from your working branch without interfering with regular development. The script is available in Python.

Installation

Download the script and make it executable:

curl -o ~/bin/git-store-file https://raw.githubusercontent.com/doitian/dotfiles-public/master/default/bin/git-store-file
chmod +x ~/bin/git-store-file

Usage

The script has four main commands: store, status, restore, and ls.

Store Files

Store one or more files to the remote branch, including files that are ignored by Git:

# Store a single file
git-store-file config.local.json
# Store multiple files
git-store-file config.local.json secrets.env
# Use a custom branch name
git-store-file --branch backup config.local.json
# Use a custom remote
git-store-file --remote upstream config.local.json

When you run the store command, the script will:

Create a temporary Git index to avoid conflicts with your working directory
Load the target branch’s current state into the temporary index
Add the specified files using git add -f to include ignored files
Create a commit with the changes
Display commit statistics and prompt for confirmation
Push the commit to the remote branch

Check Status

Check which files stored in the remote branch have been modified in your local working directory:

# Check status
git-store-file status
# Show diff for modified files
git-store-file status --diff

Restore Files

Restore files from the remote branch to your working directory:

# Restore a specific file
git-store-file restore config.local.json
# Restore all files from the branch
git-store-file restore

List Files

List all files stored in the remote branch:

git-store-file ls

Options

-b, --branch BRANCH: Specify the branch name (default: _store)
-r, --remote REMOTE: Specify the remote name (default: origin)
-h, --help: Show help message

Examples

# Store local configuration
git-store-file .env.local
# Check what's changed
git-store-file status -d
# Restore after cloning
git-store-file restore .env.local
# List all backed up files
git-store-file ls

How It Works

The script uses Git’s low-level plumbing commands to manipulate the repository without affecting your working directory:

Temporary Index: Uses the GIT_INDEX_FILE environment variable to create a temporary index, completely isolated from your working directory
Read Tree: Loads the target branch’s tree structure into the temporary index
Force Add: Uses git add -f to add files even if they’re listed in .gitignore
Commit Tree: Creates a commit object directly using git commit-tree, bypassing the normal commit workflow
Direct Push: Pushes the commit hash directly to the remote branch reference

This low-level approach ensures that:

Your working directory remains unchanged
The main branch is never affected
Ignored files can be stored without modifying .gitignore
The backup branch maintains full Git history

Conclusion

The git-store-file script solves the problem of backing up ignored files in a clean, Git-native way. By leveraging a remote branch, it keeps everything in one repository while maintaining clear separation between your main codebase and backup files. Whether you’re managing local configurations, IDE settings, or personal development scripts, this tool provides a simple and reliable backup solution that integrates seamlessly with your existing Git workflow.

How to Activate mise for Cursor When It Runs Shell Commands

me@iany.me (Ian Yang) — Wed, 26 Nov 2025 11:31:37 +0800

When using mise with Cursor, you may notice that the mise environment is not activated when Cursor executes shell commands. This occurs because Cursor launches a non-login shell, which initializes differently than an interactive login shell and therefore does not automatically source your usual mise setup.

The Problem

Cursor runs commands in a non-login shell, which means:

For zsh: Only .zshenv is loaded (not .zshrc)
For bash: Only .bash_profile is loaded (not .bashrc)

If you’ve configured mise activation in .zshrc or .zprofile, it won’t be available when Cursor executes commands.

The Solution

The solution is to activate mise in .zshenv (for zsh) or .bash_profile (for bash) when the file is loaded by Cursor. You can detect Cursor by checking for the CURSOR_AGENT environment variable.

Add this to your ~/.zshenv or ~/.bash_profile:

# Activate mise when running in Cursor
if [[ -n "$CURSOR_AGENT" ]]; then
eval "$(mise activate)"
fi

Why This Works

.zshenv (or .bash_profile) is always sourced for non-login shells
The CURSOR_AGENT environment variable is set when Cursor runs commands
By conditionally activating mise only when CURSOR_AGENT is present, you avoid potential conflicts or slowdowns in other non-login shell contexts
This ensures mise and all its configured tools are available when Cursor executes terminal commands

After adding this configuration, rerun Cursor chats, and mise should be available in all Cursor shell commands.

Temporary Vi Mode in PowerShell

me@iany.me (Ian Yang) — Sat, 22 Nov 2025 23:17:05 +0800

I miss the feature in readline (Bash/Zsh) where Ctrl+x, Ctrl+v switches to Vi command mode temporarily. In that workflow, entering Insert mode switches back to Emacs mode. PSReadLine has a command ViCommandMode, but binding it directly in Emacs mode will report errors on every key input.

The solution requires handling the mode change event to toggle the global EditMode between Emacs and Vi.

When entering “Insert” mode (the default for Emacs users), we force the mode to Emacs and set up the trigger for the temporary Vi mode. When that trigger (Ctrl+x, Ctrl+v) is fired, we switch the edit mode to Vi and jump to command mode.

Here is the configuration for your $PROFILE:

function OnViModeChange {
if ($args[0] -eq 'Insert') {
Set-PSReadLineOption -EditMode Emacs
Set-PSReadlineKeyHandler -Chord "Ctrl+x,Ctrl+v" -ScriptBlock {
Set-PSReadLineOption -EditMode Vi
[Microsoft.PowerShell.PSConsoleReadLine]::ViCommandMode()
}
# Add other Emacs key bindings here.
# Switching `EditMode` resets key bindings to their defaults.
# Remember to reconfigure them.
# Set-PSReadlineKeyHandler -Chord "Ctrl+w" -Function BackwardKillWord
}
}
Set-PSReadLineOption -ViModeIndicator Script
Set-PSReadLineOption -ViModeChangeHandler $Function:OnViModeChange
Set-PSReadLineOption -EditMode Emacs
# Reuse the function to start Emacs mode
OnViModeChange Insert

Switching EditMode resets key bindings to their defaults. Remember to reconfigure them.

Study on Quotient Spaces

me@iany.me (Ian Yang) — Tue, 18 Nov 2025 21:23:25 +0800

I’m reading Linear Algebra Done Right by Axler and found the section on quotient spaces difficult to understand, so I researched and took these notes.

Definitions

3.95 notion: $v + U$

Suppose $v \in V$ and $U \subseteq V$ . Then $v + U$ is the subset of $V$ defined by

 $v + U = \{v + u : u \in U\}.$

Also called a translate. Attention that a translate is a set.

3.97 definition: translate

Suppose $v \in V$ and $U \subseteq V$ , the set $v + U$ is said to be a translate of $U$ .

Quotient space is a set of all translates (set of sets):

3.99 definition: quotient space, $V/U$

Suppose $U$ is a subspace of $V$ . Then the quotient space $V/U$ is the set of all translates of $U$ . Thus

 $V/U = \{v + U : v \in V\}.$

Quotient space is a set of sets. There are duplicates for each $v \in V$ because for some $v_1, v_2 \in V$ , $v_1 + U$ and $v_2 + U$ can be identical set.

A quotient space $V/U$ is formed by “collapsing” a subspace $U$ to zero within a larger vector space $V$ . This construction is based on an equivalence relation where two vectors $x, y \in V$ are considered equivalent if their difference lies in $U$ —that is, $x \sim y$ if and only if $x - y \in U$ . wikipedia

Lemmas

3.101 two translates of a subspace are equal or disjoint

Suppose $U$ is a subspace of $V$ and $v, w \in V$ . Then

 $v - w \in U \iff v + U = w + U \iff (v + U) \cap (w + U) \neq \emptyset$

If two translates are not disjoint (the union set is not empty), they must be equal. So they are equal or disjoint.

All distinct translates of a subspace are disjoint. Given any $v \in V$ , it belongs to only one translate.

Since the quotient space $V/U$ is a set of translates of a subspace, it is like a disjoint partition of values in $V$ . By using the definition of quotient map

3.104 definition: quotient map, $\pi$

Suppose $U$ is a subspace of $V$ . The quotient map $\pi : V \to V/U$ is the linear map defined by

 $\pi(v) = v + U$

for each $v \in V$ .

We can write that

 $\pi(v_1) = \pi(v_2) \iff v_1 - v_2 \in U$

The quotient map has two essential properties:

The null space of $\pi$ is exactly the subspace $U$ , because $v+U=0+U \iff v-0 \in U \iff v \in U$
The range of $\pi$ is the entire quotient space $V/U$

Quotient Space Is a Vector Space

First define the addition and scalar multiplication operations:

3.102 definition: addition and scalar multiplication on $V/U$

Suppose $U$ is a subspace of $V$ . Then addition and scalar multiplication are defined on $V/U$ by

 $\begin{align*} (v + U) + (w + U) &= (v + w) + U \\ \lambda(v + U) &= (\lambda v) + U \end{align*}$

for all $v, w \in V$ and $\lambda \in \mathbf{F}$ .

$v+U$ is not the unique way to represent a member in $V/U$ , because there may exist $v'\ne v$ that $u + U = v' + U$ . The operations make sense only when the choice of $v$ to represent a translate makes no differences.

Specifically, suppose $v_1, v_2, w_1, w_2 \in V$ such that

 $v_1 + U = v_2 + U \quad\textrm{and}\quad w_1 + U = w_2 + U$

From the addition definition:

 $\begin{align*} (v_1+U) + (w_1+U) &= (v_1 + w_1) + U \\ (v_2+U) + (w_2+U) &= (v_2 + w_2) + U \end{align*}$

The left side of the two equations indeed are the different representation of the same equation, so we must show that the right side equal: $(v_1 + w_1)+U=(v2+w2)+U$ .

This applies to scalar multiplication as well:

 $\begin{align*} \lambda(v_1 + U) &= (\lambda v_1) + U \\ \lambda(v_2 + U) &= (\lambda v_2) + U \end{align*}$

We must show that $(\lambda v_1) + U = (\lambda v_2) + U$ .

Dimension

The dimension of the quotient space is given by a simple subtraction, relating the dimension of $V/U$ to the “lost” dimension of $U$ :

3.105 dimension of quotient space

Suppose $V$ is finite-dimensional and $U$ is a subspace of V. Then

 $\text{dim } V/U = \text{dim }V - \text{dim }U.$

Linear Map from V/(null T) to W

3.106 notation: $\widetilde{T}$

Suppose $T \in \mathcal{L}(V, W)$ . Define $\widetilde{T}: V/(\text{null } T) \to W$ by

 $\widetilde{T}(v + \text{null } T) = Tv.$

Think of merging inputs having the same output. These inputs will be the same input in the quotient space $V/(\text{null } T)$ .

For any $v_1, v_2 \in V$ that $Tv_1 = Tv_2$ , $v_1 + \mathrm{null}\, T$ and $v_2 + \mathrm{null}\, T$ are the same value in $V/(\mathrm{null}\, T)$ . This makes $\widetilde{T}$ injective. Because $\mathrm{range}\,\widetilde{T}=\mathrm{range}\, T$ , $\widetilde{T}$ is also surjective on to $\mathrm{range}\, T$ .

3.63 invertibility $\iff$ injectivity and surjectivity

A linear map is invertible if and only if it is injective and surjective.

3.63 shows us that $\widetilde{T}$ is invertible, and according to the definition of isomorphic, $V/(\mathrm{null}\, T)$ and $\mathrm{range}\,T$ are isomorphic vector spaces and $\widetilde{T}$ is their isomorphism.

3.69 definition: isomorphism, isomorphic

An isomorphism is an invertible linear map.
Two vector spaces are called isomorphic if there is an isomorphism from one vector space onto the other one.

One of the key uses of $\widetilde{T}$ is demonstrating a canonical isomorphism. For any linear map $T \in \mathcal{L}(V, W)$ , the quotient space $V/(\text{null } T)$ is isomorphic to the image space $\text{range } T$ . This shows that the quotient space $V/(\text{null } T)$ serves as a way to “mod out” the non-injective part of $T$ .

Hosting Static Site on Cloudflare R2

me@iany.me (Ian Yang) — Wed, 22 Oct 2025 18:46:38 +0800

This post explains how to set up a static site on Cloudflare R2.

I recently migrated my blog to Cloudflare R2, and the process went smoothly until I encountered R2’s lack of native support for rewriting URLs to index.html files. This post explains how I resolved this issue.

Setting Up

Create the bucket.
Add the custom domain blog.iany.me. To redirect iany.me and www.iany.me to the blog, add these as custom domains as well. Since I host the domain in Cloudflare, the DNS record is automatically configured.

Publishing

I use rclone to deploy the site to R2. Below is the GitHub workflow I employ. Noting that the fetch-depth option for actions/checkout retrieves the complete repository history. This works with Hugo’s --enableGitInfo flag to accurately determine article creation dates.

name: Deploy Hugo Site to Cloudflare R2
on:
push:
branches:
- master
workflow_dispatch:
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install latest Hugo
run: |
TAG=$(curl -s https://api.github.com/repos/gohugoio/hugo/releases/latest \
| grep '"tag_name":' \
| head -1 \
| sed -E 's/.*"([^"]+)".*/\1/')
curl -L "https://github.com/gohugoio/hugo/releases/download/${TAG}/hugo_${TAG#v}_Linux-64bit.tar.gz" \
-o hugo.tar.gz
tar -xzf hugo.tar.gz hugo
sudo mv hugo /usr/local/bin/
hugo version
- name: Install rclone
run: |
curl https://rclone.org/install.sh | sudo bash
- name: Build site with Hugo
run: hugo --minify --enableGitInfo
- name: Configure rclone for Cloudflare R2
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
run: |
rclone config create r2 s3 \
provider Cloudflare \
access_key_id "${R2_ACCESS_KEY_ID}" \
secret_access_key "${R2_SECRET_ACCESS_KEY}" \
endpoint "${R2_ENDPOINT}" \
--quiet
- name: Deploy to Cloudflare R2
env:
R2_BUCKET: ${{ secrets.R2_BUCKET }}
run: |
rclone copy public/ r2:"${R2_BUCKET}" \
--checksum \
--no-traverse \
--verbose

Rules

Go to the dashboard of the domain iany.me and go to the section Rules.

Redirect Rules

I have added 3 redirect rules:

When incoming requests have a hostname iany.me or www.iany.me, redirect to concat("https://blog.iany.me", http.request.uri.path).
Redirect https://blog.iany.me/*/index.html to https://blog.iany.me/${1}/.
Redirect http://* to https://${1}

Although the Redirect Rules manual says if multiple rules make the same modification, the last executed rule wins, I have to put rule 2 before 3.

URL Rewrite Rules

I added 2 rewrite rules for index.html:

When the request URL is https://blog.iany.me or https://blog.iany.me/, rewrite the URL path to /index.html.
When the request URL matches https://blog.iany.me/*/, rewrite the path to ${1}/index.html

Cache Rules

I added 2 cache rules for images and static content:

Set cache TTL to 1 month for png, jpg, jpeg, and svg files.
Set cache TTL to 1 year for files in /js, /fonts, /css, and /uploads.

Explain Atomic Cross-Chain Swaps by Herlihy

me@iany.me (Ian Yang) — Fri, 19 Sep 2025 22:26:46 +0800

This article explains the paper [Herlihy, 2018]¹ in simple words. In the paper, Herlihy has introduced an effective atomic cross-chain swap protocol in order to exchange assets across multiple blockchains among multiple parties.

Model

A swap can be modeled using a directed graph $(V, A)$ . $V$ is a set of involved users, and $A$ is a set of payments. Each payment is annotated as $(u, v)$ where $u$ is the payer and $v$ is the payee. The Atomic Swap Protocol ensures that when all users follow the protocol, if one payment succeeds, all other payments must succeed.

For example, following model represents a swap that Alice sends a payment to Bob, and Bob sends a payment to Carol.

 $\begin{array}{rl} V &= \{\mathrm{Alice}, \mathrm{Bob}, \mathrm{Carol}\} \\ A &= \{(\mathrm{Alice},\mathrm{Bob}), (\mathrm{Bob}, \mathrm{Carol})\} \end{array}$

Simple Protocol

When all payments can be connected into a path, like the example above, the protocol is simple and is similar to the payment hops in lightning network.

A path is annotated as $(u_1, \ldots, u_l)$ , which is consisted of $l-1$ payments: $(u_1, u_2),(u_2, u_3),\ldots,(u_{l-1}, u_l)$ .

The final payee $u_l$ generates a secret $s$ and sends its hash $H(s)$ to the payer $u_1$ . The payer $u_1$ creates a Hash Time Locked Contract $(H(s), 2(l-1)\Delta)$ that either $u_2$ can get the fund by providing the secret $s$ , or $u_1$ can get the refund when time has elapsed $2(l-1)\Delta$ units. The users $u_i$ from $u_2$ to $u_{l-2}$ creates a Hash Time Locked Contract $(H(s), 2(l-i)\Delta)$ when they confirm that they have received the inbound contract. The payee $u_l$ can settle the deal by providing the secret $s$ to $u_{l-1}$ , and each user uses the received secret $s$ to settle their inbound contract.

The simple protocol also works when payee and payer are the same user ( $u_1 = u_l$ ). From engineering points, it should work for most scenarios.

Arbitrary Connected Directed Graph

The paper also extends the swap protocol to arbitrary connected directed graph. A connected directed graph means for any two user $u, v$ , there are a payment path from $u$ to $v$ , and a payment path from $v$ to $u$ . Such directed graphs contain loops, as in the example below:

 $\begin{array}{rl} V = \{&A, B, C, D, E, F\} \\ A = \{ \\ &(A, B), (B, C), (C, A), \\ &(C, D), (D, B), \\ &(D, E), (E, F), (F, D) \\ \} \\ \end{array}$

Leaders Selection

The protocol must select a Feedback Vertex Set of the directed graph as leaders. A feedback vertex set is a subset of $V$ that once removed from the graph, there will be no loops in the graph. For example, $(B, D)$ is a feedback vertex set of the example above. When there are multiple candidates, choose an arbitrary set.

The leaders generate secrets and publish their hashes. If $(B, D)$ are selected, they should publish $(H(s_B), H(s_D))$ .

The protocol then move forward in two phases.

Phase One

In the first phase, users offer outbound contracts to counterparty for each payment.

Leaders must offer contracts first:

Publish a contract on every outbound payments, then
wait until all inbound contracts have been published.

Followers must confirm inbound contracts first:

Wait until correct inbound contracts have been published, then
publish a contract on every outbound payments.

The contracts are Hash Time Locked Contracts $(h, t)$ that payee can get the fund using the proof to unlock $h$ , or the payer gets the refund after timeout $t$ .

For a payment $(u, v)$ , the proof of $h$ is a triple $(s, p, \sigma)$ where

$p$ is any payment path $(u_0, \ldots, u_k)$ starting from the payee $v$ ( $u_0 = v$ ) to a leader $u_k$ . The path $p$ can have a length of 0 if $v$ itself is a leader.
$s$ is the secret generated by the ending leader of the path $p$ , and $h = H(s)$ .
$\sigma$ is a nested signatures of $s$ signed by users in the path $p$ .
- The innest signature is $\mathrm{sig}_k = \mathit{sig}(s, u_k)$
- For user $u_i$ ( $0 \leq i \leq k - 1$ ), the signature is $\mathrm{sig}_i = \mathit{sig}(\mathrm{sig}_{i+1}, u_i)$
- $\sigma = \mathrm{sig}_0$

The hash lock $h$ can be set to $H(s)$ for the shortest $p$ , or just the hashes of all secrets to all unlocking using arbitrary ending leader of the path $p$ .

The time lock $t$ is $(\mathit{diam}(\mathcal{D}) + |p|)\Delta$ , where

$\mathit{diam}(\mathcal{D})$ is the longest length of a payment path without duplicated users in the directed graph.
$|p|$ is the length of the path $p$ in the hash lock.
$\Delta$ is a configured time unit constant for the time out.

Let’s see two examples.

The first example is the payment from a leader: $(B, C)$ .

There are three candidate path $p$ from the payee $C$ to a leader either $B$ or $D$ :

$(C, A, B)$
$(C, D)$
$(C, D, B)$

$B$ can choose any one, but the shortest one is recommended. Let’s use $(C, D)$ . $B$ can publish the contract because he/she is the leader.

The hash lock for $(B, C)$ is $H(s_D)$ .
The time lock is $6\Delta$ , because the length of the longest path $(A, B, C, D, E, F)$ is 5, and the length of $(C, D)$ is 1.

The second example is the payment from a follower: $(C, A)$ .

$C$ can choose the path $(A, B)$ and publish the contract:

The hash lock is $H(s_B)$ .
The time lock is $6\Delta$

Phase Two

In phase two, leaders can unlock the inbound contracts because they know all secrets. For example, the leader $B$ can unlock the inbound payment $(A, B)$ by offering:

The path $p = (B)$
The secret $s_B$
The signature $sig(s_B, B)$

Followers must trace outbound contracts unlocking events. For example, $A$ can unlock the contract for $(C, A)$ after receiving the unlocking event above:

The path $p = (A, B)$
The secret $s_B$ since B has published it when unlocking $(A, B)$ .
The signature $sig(sig(s_B, B), A)$

Herlihy, M. (2018). Atomic Cross-Chain Swaps (No. arXiv:1801.09515). arXiv. https://doi.org/10.48550/arXiv.1801.09515

Envrc Alternative for PowerShell in Windows

me@iany.me (Ian Yang) — Sat, 06 Apr 2024 08:55:34 +0800

This post introduces a solution for automatically setting up and tearing down shell environments for PowerShell in Windows. It is proposed as a potential alternative to the bash-based tool direnv, which, while effective at loading .envrc files in the current or nearest ancestor directory, has limited compatibility with PowerShell in Windows.

.envrc.ps1

The basic idea is creating a file .envrc.ps1 with two entries: setup and teardown. The setup entry is responsible for setting up the environment up entering the directory, and the teardown entry is used when leaving it.

PowerShell allows code to be executed from a file using the dot command. This means that the entire file can be used as the setup entry. For the teardown entry, a function can be defined within the same file, and I use the function name down.

The layout of the file .envrc.ps1 looks like:

# setup
$env:GH_TOKEN = "secret"
# teardown
function global:down {
$env:GH_TOKEN = ""
}

The global: scope before the function name is necessary when this file is loaded in the hook, as I will do later.

The corresponding commands for setup and teardown entries:

Setup: . .envrc.ps1
Teardown: down

Hook

To hook the setup and teardown entries, I reference this example.

using namespace System;
using namespace System.Management.Automation;
$hook = [EventHandler[LocationChangedEventArgs]] {
param([object] $source, [LocationChangedEventArgs] $eventArgs)
end {
# 1. `down` for $eventArgs.OldPath
# 2. `. .envrc.ps1` for $eventArgs.NewPath
}
};
$currentAction = $ExecutionContext.SessionState.InvokeCommand.LocationChangedAction;
if ($currentAction) {
$ExecutionContext.SessionState.InvokeCommand.LocationChangedAction = [Delegate]::Combine($currentAction, $hook);
} else {
$ExecutionContext.SessionState.InvokeCommand.LocationChangedAction = $hook;
};

To run the code snippet, PowerShell 7 or later is required. You can install it from the repository.

Here is the algorithm for identifying setup and teardown entries:

Find the nearest .envrc.ps1 for the old path.
Find the nearest .envrc.ps1 for the new path.
If the two paths are the same, skip the next step.
If they differ, invoke the down function and remove it if it exists. Then, if found, load the .envrc.ps1 file for the new path.

The full code example to be added into PowerShell $PROFILE file.

using namespace System;
using namespace System.Management.Automation;
# existing PowerShell profile contents
# skip when version is less than 7
if ($PSVersionTable.PSVersion.Major -lt 7) {
return
}
# find the .envrc.ps1 in the currenct directory or the nearest ancestor directory.
function Find-NearestEnvrc {
param (
[Parameter(Mandatory=$true)]
[string]$StartDir
)
$currentDir = Resolve-Path $StartDir
while ($currentDir -ne "") {
$envrcPath = Join-Path $currentDir ".envrc.ps1"
if (Test-Path $envrcPath) {
return $envrcPath
}
$currentDir = Split-Path $currentDir -Parent
}
return ""
}
# hook the setup and teardown entries
$hook = [EventHandler[LocationChangedEventArgs]] {
param([object] $source, [LocationChangedEventArgs] $eventArgs)
end {
$oldEnvrc = Find-NearestEnvrc $eventArgs.OldPath
$newEnvrc = Find-NearestEnvrc $eventArgs.NewPath
if ($oldEnvrc -ne $newEnvrc) {
Get-Command down -ErrorAction SilentlyContinu
if (Get-Command down -ErrorAction SilentlyContinu) {
down
Remove-Item Function:down
}
if ($newEnvrc -ne "") {
. $newEnvrc
}
}
}
};
$currentAction = $ExecutionContext.SessionState.InvokeCommand.LocationChangedAction;
if ($currentAction) {
$ExecutionContext.SessionState.InvokeCommand.LocationChangedAction = [Delegate]::Combine($currentAction, $hook);
} else {
$ExecutionContext.SessionState.InvokeCommand.LocationChangedAction = $hook;
};

Example Usage

# setup
# set environment variable
$env:GH_TOKEN = "secret"
# load python virtual env
. .venv\Scripts\Activate.ps1
# add a helper function
function global:build {
cargo build
}
# teardown
function global:down {
# unset environment variable
$env:GH_TOKEN = ""
# unload python virtual env
deactivate
# remove the helper function
Remove-Item Function:build
}

How to Verify JoyID WebAuthn Signature

me@iany.me (Ian Yang) — Sun, 17 Dec 2023 17:35:33 +0800

JoyID is a multichain, cross-platform, passwordless and mnemonic-free wallet solution based on FIDO WebAuthn protocol and Nervos CKB.

This post shows how to verify the signature from the method signChallenge of the @joyid/ckb package. The method reference page has a demo. I use the demo to obtain an example response then verify the response using the OpenSSL command line and the Python library PyCryptodome.

The JoyID follows the WebAuthn specification and employs secp256r1 for signing. Although the guide references section 6.3.3 of the WebAuthn specification, titled “The authenticatorGetAssertion Operation”, I discovered that the example in this repository provided me much more helps.

The Response Parsing

This is the example I obtained from the demo.

{
"signature": "MEUCICF25qdO6nLreEoBHnyaw-9R6XFHbIu-NwsAI53t016qAiEAgmhlwTEMxoWxKj79R1rUkB_6nrhJfws82DqHkY_HnqQ",
"message": "K4sF4fAwPvuJj-TW3mARmMenuGSrvmohxzsueH4YfFIFAAAAAHsidHlwZSI6IndlYmF1dGhuLmdldCIsImNoYWxsZW5nZSI6IlUybG5iaUIwYUdseklHWnZjaUJ0WlEiLCJvcmlnaW4iOiJodHRwczovL3Rlc3RuZXQuam95aWQuZGV2IiwiY3Jvc3NPcmlnaW4iOmZhbHNlLCJvdGhlcl9rZXlzX2Nhbl9iZV9hZGRlZF9oZXJlIjoiZG8gbm90IGNvbXBhcmUgY2xpZW50RGF0YUpTT04gYWdhaW5zdCBhIHRlbXBsYXRlLiBTZWUgaHR0cHM6Ly9nb28uZ2wveWFiUGV4In0",
"challenge": "Sign this for me",
"alg": -7,
"pubkey": "3538dfd53ad93d2e0a6e7f470295dcd71057d825e1f87229e5afe2a906aa7cfc099fdfa04442dac33548b6988af8af58d2052529088f7b73ef00800f7fbcddb3",
"keyType": "main_key"
}

pubkey

The pubkey field represents the uncompressed public key concatenating two 32-byte integers in hex. PyCryptodome can import the key by prepending the flag 0x04. OpenSSL uses PEM to encode keys, and PyCryptodome can help here to export the key in PEM format.

from Crypto.PublicKey import ECC
pubkey_raw_hex = "3538dfd53ad93d2e0a6e7f470295dcd71057d825e1f87229e5afe2a906aa7cfc099fdfa04442dac33548b6988af8af58d2052529088f7b73ef00800f7fbcddb3"
pubkey = ECC.import_key(bytes.fromhex("04" + pubkey_raw_hex), curve_name="secp256r1")
with open("pubkey.pem", "wt") as pemfile:
pemfile.write(pubkey.export_key(format="PEM"))

Double check the key using OpenSSL:

$ openssl ec -text -inform PEM -in pubkey.pem -pubin
...
Public-Key: (256 bit)
pub:
04:35:38:df:d5:3a:d9:3d:2e:0a:6e:7f:47:02:95:
dc:d7:10:57:d8:25:e1:f8:72:29:e5:af:e2:a9:06:
aa:7c:fc:09:9f:df:a0:44:42:da:c3:35:48:b6:98:
8a:f8:af:58:d2:05:25:29:08:8f:7b:73:ef:00:80:
0f:7f:bc:dd:b3
ASN1 OID: prime256v1
NIST CURVE: P-256
...

message

The message is a binary encoded by base64 RFC 4648 §5 without the equal sign (=) paddings. Many base64 tools and libraries require padding equal sign (=) in the end of the string to make the length multiple of 4. The message in the example response has a length 351, which requires one = padding. A trick is always padding two equals at the end of the string before decoding.

The first 37 bytes in message are authenticator data, and the following bytes are client data in JSON.

The section section 6.1 in the WebAuthn specification defines the layout of the authenticator data.

rpIdHash, 32 bytes: the sha256 checksum of the text testnet.joyid.dev
flags, 1 byte: 0x05 in JoyID
signCount, 4 bytes: all zeros

base64 -d <<<'
K4sF4fAwPvuJj-TW3mARmMenuGSrvmohxzsueH4YfFIFAAAAAHsidHlwZSI6Indl
YmF1dGhuLmdldCIsImNoYWxsZW5nZSI6IlUybG5iaUIwYUdseklHWnZjaUJ0WlEi
LCJvcmlnaW4iOiJodHRwczovL3Rlc3RuZXQuam95aWQuZGV2IiwiY3Jvc3NPcmln
aW4iOmZhbHNlLCJvdGhlcl9rZXlzX2Nhbl9iZV9hZGRlZF9oZXJlIjoiZG8gbm90
IGNvbXBhcmUgY2xpZW50RGF0YUpTT04gYWdhaW5zdCBhIHRlbXBsYXRlLiBTZWUg
aHR0cHM6Ly9nb28uZ2wveWFiUGV4In0=' |
dd bs=1 count=37 2>/dev/null |
xxd
#=> 00000000: 2b8b 05e1 f030 3efb 898f e4d6 de60 1198
#=> 00000010: c7a7 b864 abbe 6a21 c73b 2e78 7e18 7c52
#=> 00000020: 0500 0000 00

Check the first two lines with the sha256 checksum:

echo -n 'testnet.joyid.dev' | sha256sum
#=> 2b8b05e1f0303efb898fe4d6de601198c7a7b864abbe6a21c73b2e787e187c52 -

The client data JSON looks like this:

base64 -d <<<'
K4sF4fAwPvuJj-TW3mARmMenuGSrvmohxzsueH4YfFIFAAAAAHsidHlwZSI6Indl
YmF1dGhuLmdldCIsImNoYWxsZW5nZSI6IlUybG5iaUIwYUdseklHWnZjaUJ0WlEi
LCJvcmlnaW4iOiJodHRwczovL3Rlc3RuZXQuam95aWQuZGV2IiwiY3Jvc3NPcmln
aW4iOmZhbHNlLCJvdGhlcl9rZXlzX2Nhbl9iZV9hZGRlZF9oZXJlIjoiZG8gbm90
IGNvbXBhcmUgY2xpZW50RGF0YUpTT04gYWdhaW5zdCBhIHRlbXBsYXRlLiBTZWUg
aHR0cHM6Ly9nb28uZ2wveWFiUGV4In0=' |
dd bs=1 skip=37 2>/dev/null |
jq
{
"type": "webauthn.get",
"challenge": "U2lnbiB0aGlzIGZvciBtZQ",
"origin": "https://testnet.joyid.dev",
"crossOrigin": false,
...
}

Notice the challenge field. It is the parameter passed to signChallenge, in base64.

base64 -d <<<'U2lnbiB0aGlzIGZvciBtZQ=='
#=> Sign this for me

Attention that message is not the binary to be signed. According to the Figure 4, Generating an assertion signature, in the WebAuthn specification, the binary to be signed is a concatenation of the authenticator data and the sha256 checksum of the client data JSON.

The following code shows how to prepare the message to sign and save it into the file message.bin. Attention that base64 must use the alternative keys - and _ to replace + and / respectively.

Attention

To decode base64 “RFC 4648 §5” in python, use either base64.b64decode(s, altchars="-_") or binascii.urlsafe_b64decode(s).

import base64
from Crypto.Hash import SHA256
message_bin = base64.urlsafe_b64decode(
"K4sF4fAwPvuJj-TW3mARmMenuGSrvmohxzsueH4YfFIFAAAAAHsidHlwZSI6Indl"
"YmF1dGhuLmdldCIsImNoYWxsZW5nZSI6IlUybG5iaUIwYUdseklHWnZjaUJ0WlEi"
"LCJvcmlnaW4iOiJodHRwczovL3Rlc3RuZXQuam95aWQuZGV2IiwiY3Jvc3NPcmln"
"aW4iOmZhbHNlLCJvdGhlcl9rZXlzX2Nhbl9iZV9hZGRlZF9oZXJlIjoiZG8gbm90"
"IGNvbXBhcmUgY2xpZW50RGF0YUpTT04gYWdhaW5zdCBhIHRlbXBsYXRlLiBTZWUg"
"aHR0cHM6Ly9nb28uZ2wveWFiUGV4In0==",
)
authenticator_data = message_bin[:37]
client_data = message_bin[37:]
message_to_sign = authenticator_data + SHA256.new(client_data).digest()
with open("message.bin", "wb") as fout:
fout.write(message_to_sign)

Attention

The message in the response is not the binary to be signed. Instead, the binary to be signed is a concatenation of the authenticator data and the sha256 checksum of the client data JSON.

signature

The field signature are two 32-byte integers first encoded in DER, then base64 RFC 4648 §5 without the equal sign (=) paddings.

Many base64 tools and libraries require padding equal sign (=) in the end of the string to make the length multiple of 4. The signature in the example response has a length 95, which requires one = padding.

OpenSSL also stores signature in DER, let’s save one in the file signature.der:

base64 -d <<<"MEUCICF25qdO6nLreEoBHnyaw-9R6XFHbIu-NwsAI53t016qAiEAgmhlwTEMxoWxKj79R1rUkB_6nrhJfws82DqHkY_HnqQ=" > signature.der

The command openssl asn1parse can parse the file signature.der in the DER format.

openssl asn1parse -dump -inform DER -in signature.der
# Output =>
# 0:d=0 hl=2 l= 69 cons: SEQUENCE
# 2:d=1 hl=2 l= 32 prim: INTEGER :2176E6A74EEA72EB784A011E7C9AC3EF51E971476C8BBE370B00239DEDD35EAA
# 36:d=1 hl=2 l= 33 prim: INTEGER :826865C1310CC685B12A3EFD475AD4901FFA9EB8497F0B3CD83A87918FC79EA4

PyCryptodome expects the signature of 64 bytes for two 32-byte integers. Following code uses the asn1 module to extract the raw signature from the DER binary.

import base64
from Crypto.Util.asn1 import DerSequence
signature_der = base64.urlsafe_b64decode(
"MEUCICF25qdO6nLreEoBHnyaw-9R6XFHbIu-NwsAI53t016qAiEAgmhlwTEMxoWx"
"Kj79R1rUkB_6nrhJfws82DqHkY_HnqQ=",
)
signature_seq = DerSequence()
signature_seq.decode(signature_der)
print(signature_seq[0].to_bytes(32).hex())
# => 2176e6a74eea72eb784a011e7c9ac3ef51e971476c8bbe370b00239dedd35eaa
print(signature_seq[1].to_bytes(32).hex())
# => 826865c1310cc685b12a3efd475ad4901ffa9eb8497f0b3cd83a87918fc79ea4

Verifying

PyCryptodome:

from Crypto.Hash import SHA256
from Crypto.Signature import DSS
DSS.new(pubkey, "fips-186-3").verify(SHA256.new(message_to_sign), signature)
print("Verified OK")

OpenSSL:

openssl dgst -sha256 -verify pubkey.pem -signature signature.der message.bin

Full Python code (Gist)

import base64
from Crypto.Hash import SHA256
from Crypto.PublicKey import ECC
from Crypto.Signature import DSS
from Crypto.Util.asn1 import DerSequence
response = {
"signature": "MEUCICF25qdO6nLreEoBHnyaw-9R6XFHbIu-NwsAI53t016qAiEAgmhlwTEMxoWx"
"Kj79R1rUkB_6nrhJfws82DqHkY_HnqQ",
"message": "K4sF4fAwPvuJj-TW3mARmMenuGSrvmohxzsueH4YfFIFAAAAAHsidHlwZSI6IndlYmF1dGhuLmdldCIsImNoYWxsZW5nZSI6IlUybG5iaUIwYUdseklHWnZjaUJ0WlEiLCJvcmlnaW4iOiJodHRwczovL3Rlc3RuZXQuam95aWQuZGV2IiwiY3Jvc3NPcmlnaW4iOmZhbHNlLCJvdGhlcl9rZXlzX2Nhbl9iZV9hZGRlZF9oZXJlIjoiZG8gbm90IGNvbXBhcmUgY2xpZW50RGF0YUpTT04gYWdhaW5zdCBhIHRlbXBsYXRlLiBTZWUgaHR0cHM6Ly9nb28uZ2wveWFiUGV4In0",
"challenge": "Sign this for me",
"alg": -7,
"pubkey": "3538dfd53ad93d2e0a6e7f470295dcd71057d825e1f87229e5afe2a906aa7cfc099fdfa04442dac33548b6988af8af58d2052529088f7b73ef00800f7fbcddb3",
"keyType": "main_key",
}
pubkey = ECC.import_key(
bytes.fromhex("04" + response["pubkey"]),
curve_name="secp256r1",
)
with open("pubkey.pem", "wt") as fout:
fout.write(pubkey.export_key(format="PEM"))
message_bin = base64.urlsafe_b64decode(response["message"] + "==")
authenticator_data = message_bin[:37]
client_data = message_bin[37:]
# https://github.com/duo-labs/py_webauthn/blob/master/webauthn/authentication/verify_authentication_response.py
message_to_sign = authenticator_data + SHA256.new(client_data).digest()
with open("message.bin", "wb") as fout:
fout.write(message_to_sign)
signature_der = base64.urlsafe_b64decode(response["signature"] + "==")
with open("signature.der", "wb") as fout:
fout.write(signature_der)
signature_seq = DerSequence()
signature_seq.decode(signature_der)
signature = signature_seq[0].to_bytes(32) + signature_seq[1].to_bytes(32)
DSS.new(pubkey, "fips-186-3").verify(SHA256.new(message_to_sign), signature)
print("Verified OK")

Renaming Browser Tab Names

me@iany.me (Ian Yang) — Fri, 21 Jul 2023 20:31:59 +0800

Renaming browser tab names may seem like a simple task, but it can actually be quite challenging.

Most browsers sync the tab name with the web page title. Therefore, it seems simple to set tab name by setting document.title:

document.title = "Custom Tab Name";

However, many web pages use JavaScript to alter the page title, which would override the custom name. While Object.defineProperty in JavaScript can override the property setter function, it’s not possible to access the original setter function for document.title. Fortunately, browsers sync page titles with the first <title> tag, so setting its content acts as a workaround to set the page title.

Here is the bookmark for renaming a tab. I also add %t as the token for the original page tab title. This is handy such as adding a custom prefix with Work: %t, or restoring the original title without reloading the page by using %t.

// Rename the document title using a bookmarklet.
//
// As a user,
// - Once I have renamed the tab, it should not be overwritten.
// - I can include the token %t as placeholder for the real title.
// - To restore, I can rename the tab to %t
//
// Install: Copy the code to
//
// https://caiorss.github.io/bookmarklet-maker/
//
// and generate the bookmarklet.
// Init only once
if (!("_renameTitle" in document)) {
// Force creating the title tag.
document.title = document.title || "";
const titleEl = document.getElementsByTagName("title")[0];
const titleTokenRegex = /%t/g;
// Remembers the real title
let titleWithoutRenaming = document.title;
// User set title
let titleWithRenaming = "";
// Rename the document title to v.
// If v contains the token %t, replace all occurences to the
// real title.
document._renameTitle = (v) => {
titleWithRenaming = v;
titleEl.innerText = v.replace(titleTokenRegex, titleWithoutRenaming);
};
Object.defineProperty(document, "title", {
// Other code will use document.title setter to change the title.
//
// Remember the new value as the real title but still use the
// title set by user.
set: (v) => {
titleWithoutRenaming = v;
// Once document has defined the title property, its value is
// not synchronized to the tab name.
//
// Here uses a workaround to set the title tag content.
titleEl.innerText = titleWithRenaming.replace(titleTokenRegex, v);
},
get: () => titleEl.innerText,
});
}
const title = prompt(
"Rename tab (Use token %t for original title)",
document.title
);
// title is null when user cancel the dialog.
if (title !== null) {
document._renameTitle(title);
}

Transforming Markdown to Attractive PDFs: A Guide to Using Pandoc with Xelatex

me@iany.me (Ian Yang) — Sat, 29 Apr 2023 21:43:02 +0800

I used to read lengthy and complex articles in the PDF format. Despite the abundance of PDF exporting options, the resulting files often appear unappealing. However, there is a solution. With the pandoc tool and the xelatex backend, you can transform Markdown files into aesthetically pleasing PDFs. In this tutorial, I will guide you through the steps of using pandoc with xelatex and share the lessons I’ve learned.

Installation

Ubuntu users can install the package texlive-xetex and pandoc, In macOS, I recommend a minimal installation:

brew install librsvg pandoc
brew install --cask basictex
sudo tlmgr update --self
sudo tlmgr update --all
for pkg in texliveonfly xelatex adjustbox tcolorbox collectbox ucs environ \
trimspaces titling enumitem rsfs xecjk fvextra svg transparent; do
sudo tlmgr install $pkg
done

The library librsvg is required to support SVG images.

Verify the installation by converting a simple markdown file.

echo "# Hello World" > test.md
pandoc --pdf-engine=xelatex -i test.md -o test.pdf

Tips

In LaTeX, the document class serves as the template for the basic page styles.

I’m a fan of koma-script, and I will choose between scrreprt and scrbook depending whether the file to create is more like a report or a book. I recommend try them and the standard classes like report and book, then choose the one that best suites your preferences.

To customize the document class in pandoc, simply set the variable documentclass. Other variables are available for controlling the LaTeX style. I use some of them to define my default style.

DOCUMENTCLASS="${DOCUMENTCLASS:-scrreprt}"
MAINFONT="${MAINFONT:-Helvetica}"
MONOFONT="${MONOFONT:-Cartograph CF}"
pandoc --pdf-engine=xelatex \
--variable fontsize=12pt \
--variable linestretch=1.5 \
--variable geometry=a4paper \
--variable documentclass="${DOCUMENTCLASS}" \
--variable mainfont="${MAINFONT}" \
--variable monofont="${MONOFONT}" \
-i test.md -o test.pdf

Gotchas

CJK

CJK is a collective term for the Chinese, Japanese, and Korean languages. When CJKmainfont is set, pandoc handles CJK characters with the xecjk package.

pandoc --variable CJKmainfont="Noto Serif CJK SC" ...

My recommendation is to only set CJKmainfont when needed to avoid messing up the quotation marks display. Where the article contains a mixture of English and Asian characters resulting in strange looking quotation marks '"“”‘’, consider trying the following solution.

Create a file cjk.latex to force English quotation marks.

% cjk.tex
\AtBeginDocument{%
\XeTeXcharclass`^^^^2019=0
}

Include this file via the pandoc argument -H such as pandoc -H cjk.tex.

The figure below shows the results from the same markdown file on my MacBook. However, when I test this in Ubuntu, it seems there are no such issues.

Fix CJK Quotation Marks

This is the input markdown file used in the example.

English's "straight quotations" and this’s “curly quotations”.
中文的"直引号"和“弯引号”。

Code Block Syntax Highlight

Pandoc does not wrap lines in code blocks by default. It truncated the lines which are too long to fit on the page.

To solve the problem mentioned above, I followed a solution on GitHub suggested by jannick0. It involves the creation of a file that contains the LaTeX snippets below. These snippets can then be included by using the pandoc argument -H highlighting.tex. Additionally, I added a border around the code block.

highlighting.tex

% use this file via pandoc -H highlighting.tex
\usepackage{fvextra}
\DefineVerbatimEnvironment{Highlighting}{Verbatim}{breaklines,breaknonspaceingroup,breakanywhere,frame=single,framesep=8pt,rulecolor=\color[HTML]{aaaaaa},commandchars=\\\{\}}

As I replied in the thread, this solution does not work for code blocks without setting a language. I use a Lua filter to set the language of these code blocks to “text”. To run the filter, save it in a file, say highlighting.lua, and invoke pandoc with --lua-filter=highlighting.lua

highlighting.lua

function CodeBlock(el)
if #el.classes == 0 then
el.classes[1] = 'text'
end
return el
end
return {{CodeBlock = CodeBlock}}

Disable Figures Floating

The last issue to address is the floating of figures in LaTeX, which moves the images to avoid large blank space in pages. While LaTeX attempts to place figures in appropriate locations, authors may refer to them as “above” or “below” in the markdown files. This can be confusing for readers who may find that the referenced chart or diagram is actually on the following page in the PDF file.

To solve the issue, simply disable floating figures. Create a file disable-floating.tex and invoke pandoc with arguments -H disable-floating.tex.

disable-floating.tex

% https://stackoverflow.com/a/58840456/667158
\usepackage{float}
\let\origfigure\figure
\let\endorigfigure\endfigure
\renewenvironment{figure}[1][2] {
\expandafter\origfigure\expandafter[H]
} {
\endorigfigure
}

Putting It All Together

See the repo doitian/pandoc-ide.

The Ultimate Guide to Customizing Obsidian Vim Mode via QuickAdd

me@iany.me (Ian Yang) — Thu, 27 Apr 2023 21:22:30 +0800

Obsidian is my go-to note-taking app because of its availability on all desktop and mobile platforms with the bonus of Vim mode. In this guide, I’ll show you how I customize the Vim mode in Obsidian to maximize my note-taking efficiency.

Setup User Launch Scripts via QuickAdd

To customize the Vim mode in Obsidian, JavaScript is the only solution. Obsidian allows running user scripts by implementing an plugin. I use an easy alternative to run a JavaScript file on launch via QuickAdd.

First, let’s create the JavaScript file in the vault. Save the skeleton below as the file vimrc.js in any folder in the vault. Keep in mind that Obsidian does not support editing JavaScript files, so use an external editor.

vimrc.js

function notice(text) {
new Notice(text);
return text;
}
module.exports = async function (context) {
const vim = window.CodeMirrorAdapter?.Vim;
if (vim === undefined) {
new Notice(`🔴error: vim mode is disabled`);
return;
}
// Add customizations here.
console.log(`🔵info: vimrc loaded`);
};

Then, install QuickAdd by browsing the community plugins in Settings and follow the instructions on the Obsidian help page Community Plugins.

Finally, include the script in a QuickAdd macro and activate the option to execute the macro upon the plugin load.

Open the tab QuickAdd in Settings.
Click the button “Manage Macros”. In the pop-up dialog, fill in the macro name “vimrc” and click the button “Add macro“ to add a new macro vimrc.
Toggle the option “Run on plugin load” on under the newly created macro.
Configure the macro vimrc. Find the script file vimrc in the “User Scripts” section and add the script as the only step in the macro.

Attention that the script runs once on launch or reload. Running “Reload app without saving” via Command Palette will reload the file after changes,

Vim Mode Customization Techniques

For the best tutorial on customizing Vim mode and what methods the vim object provides, refer to how CodeMirror establishes the Vim mode. I will only introduce the methods that I use in my own customizations.

`vim.defineEx`

The defineEx method enables users to execute a new ex command in Vim by typing : in normal mode. For instance, the ex command :w saves the file.

I introduce defineEx first because I need an ex command to run any Obsidian command so that I can use it to add key bindings later. I name this command obr, which is the abbreviation of OBsidian Run. The command requires the Obsidian command identifier as the argument, for example, :obr app:open-settings will open the Settings dialog.

vim.defineEx("obr", "", function (cm, params) {
if (params?.args?.length !== 1) {
throw new Error(notice("🔴error: obr requires exactly 1 parameter"));
}
const command = params.args[0];
context.app.commands.executeCommandById(command);
});

To find the command identifier, I add another ex command :obl to display a list of identifiers and copy the selected one. It accepts an argument to filter the result. For example, obl open will list identifiers that contain the word “open”.

vim.defineEx("obl", "", async function (cm, params) {
let commands = Object.keys(context.app.commands.commands);
for (const keyword of params?.args ?? []) {
commands = commands.filter((command) => command.includes(keyword));
}
const choice = await context.quickAddApi.suggester(commands, commands);
if (choice !== null) {
await context.quickAddApi.utility.setClipboard(choice);
new Notice(`🔵info: copied ${choice} to the clipboard`);
}
});

`vim.map`

The vim.map method takes two arguments, lhs and rhs, which can both be either a key sequence or an ex command. The meaning of the method varies depending on the arguments passed.

When both lhs and rhs are key sequence, it maps the key sequence lhs to rhs. For example, vim.map('D', 'dd') will redefine D to delete the whole line.
When both lhs and rhs are ex commands, it creates a new ex command alias. For example, vim.map(':Reload', ':obr app:reload') adds the alias :Reload to run the Obsidian command app:reload.
When lhs is a key sequence, and rhs is an ex command, it maps the key sequence to run the ex command. For example vim.map('ZZ', ':obr app:reload') allows reloading Obsidian by typing ZZ.
When lhs is an ex command, and rhs is a key sequence, executing the ex command has the same effect as typing the key sequence.

I have a bunch of mappings from key sequences to :obr, for example, the z-family to operate on folds:

vim.map("zo", ":obr editor:toggle-fold");
vim.map("zc", ":obr editor:toggle-fold");
vim.map("za", ":obr editor:toggle-fold");
vim.map("zR", ":obr editor:unfold-all");
vim.map("zM", ":obr editor:fold-all");

To use the space key as a key mapping prefix, it must first be unmapped.

vim.unmap("<Space>");
vim.map("<Space><Space>", ":obr switcher:open");
vim.map("<Space>n", ":nohl");

`vim.defineAction` and `vim.defineOperator`

We can use both action and operator in vim.mapCommand. The difference is that an operator performs upon text objects. See how CodeMirror defines the built-in actions and operators.

I have an action swapLine to move lines around. It’s not perfect, since undo will reverse one step only.

vim.defineAction("swapLine", function(_cm, { repeat, down }) {
const command = down ? "editor:swap-line-down" : "editor:swap-line-up";
for (let i = 0; i < repeat; i++) {
context.app.commands.executeCommandById(command);
}
});
// undo after 5]e will only swap one line up.
vim.mapCommand("]e", "action", "swapLine", { down: true });
vim.mapCommand("[e", "action", "swapLine", { down: false });

The example for vim.defineOperator is titleCase, which formats the text in the APA Title Case Capitalization. For instance, gzap formats the current paragraph in the title case.

function titleCase(str, options) {
// See https://github.com/words/ap-style-title-case/blob/master/index.js how to format str to apa title case
}
// Following helper funnctions are borrowed from https://codemirror.net/5/keymap/vim.js
function cursorIsBefore(cur1, cur2) {
if (cur1.line < cur2.line) {
return true;
}
if (cur1.line == cur2.line && cur1.ch < cur2.ch) {
return true;
}
return false;
}
function cursorMin(cur1, cur2) {
return cursorIsBefore(cur1, cur2) ? cur1 : cur2;
}
function findFirstNonWhiteSpaceCharacter(text) {
if (!text) {
return 0;
}
var firstNonWS = text.search(/\S/);
return firstNonWS == -1 ? text.length : firstNonWS;
}
vim.defineOperator("titleCase", function(cm, args, ranges, oldAnchor, newHead) {
const selections = cm.getSelections();
const newSelections = selections.map((s) =>
titleCase(s, { keepSpaces: true })
);
cm.replaceSelections(newSelections);
if (args.shouldMoveCursor) {
return newHead;
} else if (
!cm.state.vim.visualMode &&
args.linewise &&
ranges[0].anchor.line + 1 == ranges[0].head.line
) {
return {
line: oldAnchor.line,
ch: findFirstNonWhiteSpaceCharacter(cm.getLine(oldAnchor.line)),
};
} else if (args.linewise) {
return oldAnchor;
} else {
return cursorMin(ranges[0].anchor, ranges[0].head);
}
return newHead;
});
vim.mapCommand("gz", "operator", "titleCase");

Practice Autofocus Method in Todoist

me@iany.me (Ian Yang) — Mon, 03 Oct 2022 19:26:38 +0800

I read the article Autofocus: The Productivity System That Treats Your to-Do List Like a River¹ recently. I used to fill the time slots with tasks daily, but I rarely completed all the planned tasks. Even for the completed ones, I often missed the scheduled time slot. I felt stressed and guilty. So I decide to give the Autofocus Method a try.

The River Filter

I will continue using Todoist. I created a filter named River. The tasks are sorted by added date. I added the filter to the sidebar and set it as my home view for easy access.

The filter has three sections:

Overdue tasks: overdue
Tasks I am doing: today | @now💧
Tasks in River: !##⏰ Events & !overdue & !today & !@now💧 & !@someday🪨

Todoist supports filter sections by separating queries with comma (,). So here is the final filter query:

overdue, today | @now💧, !##⏰ Events & !overdue & !today & !@now💧 & !@someday🪨

Section 1: Overdue Tasks

This section allows me to re-schedule the overdue tasks.

I decided to not totally abandon the due dates. However, I use them with great moderation. I set a due date if I must complete the task on that date, and before the day comes, I can forget the task.

I add such tasks to the project ⏰ Events. They will appear in the River filter when they are due or overdue.

Section 2: Tasks I Am Doing

This section lists tasks that I am doing now.

There are two kinds of such tasks.

The first is the scheduled tasks that are due today.

The second is the selected tasks from the river. I add the tag @now💧 to the task if I feel like doing it.

Section 3: Tasks in River

This section contains the tasks in the river. It excludes tasks from the project ⏰ Events, and the dismissed tasks. I tag dismissed tasks with @someday🪨 .

The Workflow

I add new tasks to the inbox without triaging the projects, tags, due dates, and priorities. Since the filter

When I have time to do something, I open the River filter and scan the tasks from top to bottom without scrolling down.

First, I re-schedule the overdue tasks.

Then, I do a quick scan on the second section—the tasks that I planned before. If I feel like doing a task in the list, I start working on it.

If I failed to find a task to work on in the previous step, I continue the scan to the last section—the tasks in the river. I add tag @now💧 to pop the task to the top of the list.

If there are no items stands out for me, I have three choices to remove tasks from the first screen:

Defer the scheduled tasks into future.
Remove the tag @now💧 from the task.
Dismiss a task by adding the tag @someday🪨.

The Autofocus Method suggests to cross the item off the list and re-enter it at the end of the list if I haven’t finished it yet. In Todoist, I duplicate the task and cross the original item off.

Here is my setup to practice the Autofocus Method in Todoist. I’m still experimenting to see whether it can improve my productivity.

References

McKay, B., & McKay, K. (2022, September 20). Autofocus: The Productivity System That Treats Your To-Do List Like a River. The Art of Manliness. https://www.artofmanliness.com/character/behavior/autofocus-the-productivity-system-that-treats-your-to-do-list-like-a-river/

Set the Minimum Width of the Active Pane in Obsidian

me@iany.me (Ian Yang) — Fri, 23 Sep 2022 16:44:51 +0800

Updates

I have created an plugin for this feature.

A simple CSS rule sets the minimum width of the active pane in Obsidian.

min-width.css

.mod-root .mod-active {
min-width: min(88%, 40rem);
}

It sets the minimum width to either 40 columns or 88% of the whole editing area, depending on which is smaller. Obsidian will auto resize the active pane to ensure it is wider enough and shrink other panes accordingly.

The snippet does not work in horizontal splits where panes stack vertically. See the chapter JavaScript Helper below how to work around it.

How to Add CSS Snippets in Obsidian

Save the CSS snippet as file .obsidian/snippets/min-width.css in the vault. Create the folder .obsidian/snippets if it does not exist.

Now the file should appear in the Appearance settings page. Toggle the switcher to enable the CSS snippet in the file.

Enable CSS Snippet in Obsidian

JavaScript Helper

CSS is useless when it can not select the target elements. To make the snippet work in horizontal splits, the CSS rule has to select the .mod-horizontal node when it has a .mod-active child. However, the pseudo-class selector :has()¹ is not available in Obsidian yet.

JavaScript can help to add classes and attributes to HTML elements.

I also want to make some panes wider, such as Excalibrain². The JavaScript code can also set the view type to the proper elements. The view type of the Excalibrain pane is excalidraw.

First, update the CSS snippet to make Excalibrain panes wider. Indeed, it makes all Excalidraw panes wider.

min-width.css

.mod-root .mod-active {
min-width: min(88%, 40rem);
}
.mod-root .mod-active[data-type="excalidraw"] {
min-width: min(88%, 60rem);
}

The last, let Obsidian run some code when the active pane has changed. I use QuickAdd³ to manage the scripts. Save the following code snippet as min-width.js in the vault.

min-width.js

const DATA_TYPE = "data-type";
function setOrRemoveDataType(el, dataType) {
if (dataType !== undefined) {
el.setAttribute(DATA_TYPE, dataType);
} else {
el.removeAttribute(DATA_TYPE);
}
}
module.exports = async function ({ app }) {
app.workspace.on("active-leaf-change", (leaf) => {
// clear .mod-active on .mod-horizontal
for (const el of app.workspace.containerEl.getElementsByClassName(
"mod-active mod-horizontal"
)) {
el.classList.remove("mod-active");
}
// bubble up data-type
const dataType = leaf.containerEl
.getElementsByClassName("workspace-leaf-content")[0]
?.getAttribute(DATA_TYPE);
setOrRemoveDataType(leaf.containerEl, dataType);
// add .mod-active and data-type to current horizontal split container
const parentNode = leaf.containerEl.parentNode;
if (parentNode.classList.contains("mod-horizontal")) {
parentNode.classList.add("mod-active");
setOrRemoveDataType(parentNode, dataType);
}
});
};

Then run it when Obsidian starts by following these steps:

Click the button “Manage Macros” in the QuickAdd settings.
Add a new macro and add the user script “min-width” to the macro.
Save the macro and check the option “Run on plugin load”.

Run JavaScript Code on Start With QuickAdd

That’s it. Restart Obsidian and the code should be effective. See how it works in the screen recording below.

Set the Minimum Width of the Active Pane in Obsidian

There are two extra plugins used in the video, Snippet Commands⁴ and Cycle through Panes.⁵

Similar Work

This work is inspired by the Vim option winwidth, one of my favorite feature of Vim.

The awesome plugin Sliding Panes⁶ provides the similar feature for Obsidian. However, it has three problems that I cannot stand.

The first, when the option “Leaf Width on Desktop/Mobile” is wider than the whole editing area, I have to scroll horizontally to see the current active pane. Although the plugin has separated the options for Desktop and Mobile, but different devices have different screen sizes. What’s worser, the app window can be any size on Desktop. The option must be less than the smallest screen width when Obsidian Sync is in use, which is not a good choice for wider devices.

The second, the plugin does not play well with horizontal splits.

The last, I really want to set the different minimum widths for different windows.

References

MDN. (2016, October 27). :has() - CSS: Cascading Style Sheets. MDN. https://developer.mozilla.org/en-US/docs/Web/CSS/:has
zsviczian. (2022, May). Excalibrain: A Graph View to Navigate Your Obsidian Vault. GitHub. https://github.com/zsviczian/excalibrain
chhoumann. (2021, June). QuickAdd for Obsidian. GitHub. https://github.com/chhoumann/quickadd
deathau. (2021, October). Snippet Commands Obsidian Plugin. GitHub. https://github.com/deathau/snippet-commands-obsidian
phibr0. (2020, November). Cycle Through Panes Obsidian Plugin. GitHub. https://github.com/phibr0/cycle-through-panes
deathau. (2020, October). Sliding Panes (Andy Matuschak Mode) Obsidian Plugin. GitHub. https://github.com/deathau/sliding-panes-obsidian

Use Italic Font for Coding

me@iany.me (Ian Yang) — Sat, 11 Sep 2021 21:21:30 +0800

I discovered Noctis theme recently and found that the font Cartograph CF is really interesting.

Cartograph is a handsome monospaced font family featuring lush italics, code-friendly ligatures

It’s funny to see those handwriting like italic characters while coding, so I purchased the font and integrated it into my tools.

For Visual Studio Code, I use Noctis. But it uses too much italic style, which decreases the readability. Here are my customizations:

{
"editor.tokenColorCustomizations": {
"[Noctis]": {
"textMateRules": [
{
"scope": "markup.list",
"settings": { "fontStyle": "" }
},
{
"scope": "text.markdown.notes.tag",
"settings": { "fontStyle": "italic" }
}
]
}
},
// ...
}

Visual Studio Code

It’s handy to inspect the syntax highlight scope under cursor using the command “Developer: Inspect Editor Tokens and Scopes”.

In macOS Terminal, I can use the escape code \e[3m to enable italic and \e[23m to reset.

printf "\e[3mitalic\e[23m"

Or using echo

echo "^[[3mitalic^[[23m"

Where the ^[ is indeed how the terminal displays Esc. One way to insert Esc is via Ctrl-v Esc.

I switched to starship recently, so I don’t need to bother setting up the italic fonts in my shell prompt by myself. Here is my starship config file:

format = "[$all](italic)"
[directory]
format = '[%\(4~|%-1~/…/%2~|%~\)]($style)[$read_only]($read_only_style) '
[git_branch]
symbol = "±"
style = "purple bold italic"
[status]
disabled = false
[nodejs]
symbol = '🤖 '

I changed the nodejs symbol, because Cartograph CF has no the nerd font icons.

I’m a heavy user of tmux, it’s tricky to enable italics in tmux. The secret is setting default-terminal to tmux.

set -g default-terminal "tmux"

Then set back the terminal to xterm-256color via default-command because macOS Terminal does not understand what is the tmux terminal.

set -g default-command "env TERM=xterm-256color zsh"

I only configure the session name to be italic.

set -g status-left "#[fg=white,bg=colour31,nobold,italics] #S "

I cannot find a cross-platform way to show italic font in vim, so I manually enable it when I want to see the funny italic characters in the comment:

function! s:Italic(enable)
if a:enable
hi Comment cterm=italic gui=italic
hi Folded cterm=italic gui=italic
let &t_ZH = "\e[3m"
let &t_ZR = "\e[23m"
else
hi Comment cterm=none gui=none
hi Folded cterm=none gui=none
let &t_ZH = ''
let &t_ZR = ''
endif
endfunction
command! ItalicEnable call s:Italic(1)
command! ItalicDisable call s:Italic(0)

Shell, Tmux and Vim

The last tool is Obsidian, I use the CSS snippet to customize the appearance:

:root {
--font-monospace: 'Cartograph CF', monospace;
}
.cm-meta, .cm-hmd-frontmatter, .tag, .image-embed[alt]:after {
font-style: italic;
font-family: var(--font-monospace);
}
.cm-meta.cm-formatting-task {
font-style: normal;
font-family: var(--default-font);
}

Fortunately, it also works on iOS after installing the fonts to the system via Fontcase.

I have also tested several frequently used iOS apps.

Textastic can use the fonts installed by Fontcase, and Cartograph CF works perfect. The app a-Shell is also fine to use the self installed fonts.

But Blink does not support system fonts. Although it can install fonts via CSS, I cannot make Blink display Cartograph CF using the correct line height. It also makes Blink sluggish and unstable.

I use an Android phone currently, but I give up the joy to use Cartograph CF on my phone. Because I don’t use it for work and it seems hard to install fonts on Android. I tried to make it work in Obsidian, by copying fonts to Obsidian vault and using remote font URL, but both failed.

SSH Authentication Using a YubiKey on Windows And the OpenSSH Client

me@iany.me (Ian Yang) — Tue, 09 Feb 2021 20:08:23 +0800

As mentioned in ♯ Yubico for Windows, I used PuTTY/Plink instead of the OpenSSH client together with YubiKey because the OpenSSH Client does not support the socket file created by GnuPG.

Plink does not work well in Windows Terminal. The Visual Studio Code SSH Remote does not support Plink as well, because it will pass some command line arguments that are not supported by Plink.

So I decide to switch back to the OpenSSH client. Fortunately, the utility wsl-ssh-pageant can create a tunnel between a Windows pipe and the pageant socket, and the OpenSSH client can use the Windows pipe as SSH_AUTH_SOCK. This article is a tutorial to set up wsl-ssh-pageant.

Enable PuTTY Support

First edit gpg-agent.conf in folder $(scoop prefix gpg)\home\, enable PuTTY support

enable-putty-support

Restart gpg agent to reload the config file.

gpg-connect-agent killagent /bye

Install OpenSSH Client

This section is copied from Installation of OpenSSH For Windows Server | Microsoft Docs

To install OpenSSH, start Settings then go to Apps > Apps and Features > Manage Optional Features. Or install it using PowerShell:

Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'
# This should return the following output:
Name : OpenSSH.Client~~~~0.0.1.0
State : NotPresent
Name : OpenSSH.Server~~~~0.0.1.0
State : NotPresent

Then, install the client feature:

# Install the OpenSSH Client
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
# It should return the following output:
Path :
Online : True
RestartNeeded : False

Set Up wsl-ssh-pageant

Install wsl-ssh-pageant by downloading the binary from the GitHub release or using scoop

scoop install wsl-ssh-pageant

Create a cmd file winssh-agent.cmd

wsl-ssh-pageant.exe --systray --winssh ssh-pageant
PAUSE

Double-clicking this file will start the tunnel.

Add the environment variable to tell OpenSSH client to use the pipe created by wsl-ssh-pageant.

[Environment]::SetEnvironmentVariable('SSH_AUTH_SOCK', '\\.\pipe\ssh-pageant', 'User')

The environment variable is only effective in new windows, so start a new terminal window to try that the tunnel works.

First, start the gpg agent and check that the card reader works:

gpg --card-status

Now try to authenticate SSH, for example, to GitHub

ssh git@github.com

Auto Start wsl-ssh-pageant

Create a shortcut in Windows Start Menu to auto-start wsl-ssh-pageant on login and allow launch it by searching in the start menu by pressing the Windows key.

# Change the path to the cmd file.
$SSHAgentLocation = "X:\Path\to\winssh-agent.cmd"
$SSHAgentShortcut = "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\Startup\SSH Agent.lnk"
$WScriptShell = New-Object -ComObject WScript.Shell
$Shortcut = $WScriptShell.CreateShortcut($SSHAgentShortcut)
$Shortcut.TargetPath = $SSHAgentLocation
$Shortcut.WindowStyle = 7
$Shortcut.Save()
mkdir -Fo "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\SSH Agent"
cp -Fo "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\Startup\SSH Agent.lnk" "$env:APPDATA\Microsoft\Windows\Start Menu\Programs\SSH Agent\SSH Agent.lnk"

Install macOS Big Sur in 2021

me@iany.me (Ian Yang) — Sat, 06 Feb 2021 20:06:11 +0800

My MacBook’s battery capacity is significantly reduced and it has many weird issues so I decide to do a clean installation of macOS Big Sur. This article is a reference for my own reference in case I need to do it again.

Preparation

I plan to erase the disk and install a clean copy of macOS, so the first step is creating the bootable installer.

Back up the data. Follow these instructions if I want to sell, give away, or trade in my Mac.
Get Big Sur from the App Store. This will install the app Install macOS Big Sur.app into /Applications.
Insert the USB stick which has at least 8GB capacity. Assuming the stick name is MyVolume, create the installer using the USB stick with following command ¹
```
sudo /Applications/Install\ macOS\ Big\ Sur.app/Contents/Resources/createinstallmedia --downloadassets --volume /Volumes/MyVolume
```
The option --downloadassets is optional but is recommended, because without proxy, it is slow to download resources during the installating stage in some regions.
Reboot the laptop into recovery mode by holding Command+R. Choose Utilities > Startup Security Utility from the menu bar and allow booting from external media.² This step is very important and must be done before erasing the disk because it requires verifying the admin user password in the installed system. I fogot it and had to recover the factory macOS version to just create a user so that I can allow booting from external media.
Ensure there’s a working Internet connection. The installation requires network to do verification.

Install

Keep the USB stick inserted. Reboot the laptop by holding Option (⌥) this time. This will bring the startup disk selection interface. Select the USB stick, which is shown as “Install Big Sur”.

This will boot into the Big Sur recovery mode.

Use disk utility to erase the disk. Use security options to securely erase the data from the disk if I plan to sell the Mac or send it to the genius bar.
Quit disk utility and choose to install Big Sur from the menu.

Gotchas to Publish Rust Crates in a Workspace

me@iany.me (Ian Yang) — Sun, 11 Oct 2020 12:03:32 +0800

It is recommended to split a huge Rust project into crates and manage them in a workspace. I’m currently working on a project which consists of about 60 crates. It works well so far until I try to publish these crates to crates.io.

I will list the problems I have met and solutions or workarounds I have adopted.

Update Cargo.toml

Crates.io has some requirements on Cargo.toml files:

Fields like license, authors, description, homepage, and repository are required.
Local crates via path must specify the version as well so the dependencies can be resolved via crates.io.

The command cargo publish --dry-run will verify whether the file Cargo.toml is valid.

There’s No `cargo publish --all`

Cargo does not support publishing all local packages yet. It means that I have to cd into each crate and publish them separately.

There are two major issues here.

The first, if a crate foo depends on bar, a version of bar satisfying foo’s requirement must be available in crates.io before publishing foo. In the project I’m working on, all crates share the same version, and they lock the local dependencies using the exact version such as bar = "= 0.0.1". If I want to publish these crates, I must topologically sort them by dependencies first and publish them in that order. The newly published crate has a delay of up to 10 seconds before it is searchable. In my publish script, I’ll retry several times when cargo complaints that it fails to resolve a dependency.

Second, it is not atomic to publish the whole workspace. If I successfully published bar 0.0.1 and later fails to publish foo 0.0.1 because of a bug in bar 0.0.1, I have to yank bar 0.0.1, bump the version, and re-publish both foo and bar.

Cyclic Dependencies

The dev-dependencies may introduce cyclic dependencies. Take the example again that the crate foo depends on bar. This time the test cases of bar depend on foo. Cargo can resolve these cyclic dependencies because it does not need the dev-dependencies to build both foo and bar, so it can build bar, foo, and then bar test cases in order.

But cargo publish requires both dependencies and dev-dependencies are available in crates.io, now cyclic dependencies will cause problems.

It’s tedious to arrange the crates to resolve the cyclic dependencies, so I adopted this workaround to remove all the dev-dependencies before publish and run cargo publish --allow-dirty to ignore the dirty git working directory.

It Is Very Slow to Publish

Finally, it is really slow to publish 60 crates in a big project. It seems that they will not share the target directory and each crate uses its own directory inside target/publish. If the workspace has three crates, foo, bar, baz, where foo depends on bar, and bar depends on baz, cargo will take quadratic time to publish all the crates:

Build baz and publish baz.
Build baz and bar, then publish bar.
Build baz, bar, and foo, then publish foo.

The target/publish also takes a lot of disk space. After I published 60 crates, the folder occupied about 80G storage. If the publish host has limited disk space, the script must clean up the folder regularly.

Monero Dynamic Block Weight

me@iany.me (Ian Yang) — Sat, 05 Sep 2020 17:52:10 +0800

Monero dynamic block weight is an interesting design. Instead of a fixed supply of transaction space, Monero uses the history blocks to determine the dynamic limit and use the median to control the increasing pace.

The major reference of this article is chapter 7.3 Money supply in the book Zero to Monero: Second Edition.

Block Weight Limit

The current block’s cumulative_weights_median is the base limit for the next block. The max block weight of the next block is 2 ∗ cumulative_weights_median and the block reward decays when the block weight is greater than cumulative_weights_median.

Block Reward decays when the weight is greater than cumulative_weights_median

The definition of cumulative_weights_median is

cumulative_weights_median = max{
300kB,
min{
max{
300kB,
median_100blocks_weights
},
50 ∗ effective_longterm_median
}
}

The item 300kB is the lower bound of cumulative_weights_median, which can be removed to simplify the formula.

The term effective_longterm_median changes slowly and can be considered as a constant here. It acts as the upper bound.

The value of cumulative_weights_median indeed is the median of the last 100 blocks weights clamped between 300kB and 50 * effective_longterm_median. In turn, the block weight is restricted by cumulative_weights_median.

The base limit cumulative_weights_median increases when there are at least 50 in the recent 100 blocks which weight is greater than cumulative_weights_median, indicating that at least half of the miners are willing to take the block reward penalty. Miners are incentivized to take the penalty because the transaction fees can cover the loss.

If the network runs at full load, cumulative_weights_median doubles every 50 blocks and will reach the upper bound 50 * effective_longterm_median eventually.

Pay attention that cumulative_weights_median falls immediately when the network load drops. If there are 50 in the last 100 blocks which weight is less than 300kB, cumulative_weights_median will become 300kB. Maintaining cumulative_weights_median at a value higher than 300kB requires continuous transaction traffic.

An example of how cumulative_weights_median changes

Long Term Limit

The value of effective_longterm_median caps the max block weight in the short term. As the name suggests, it is the median of the longterm_block_weight of the last 100,000 blocks.

effective_longterm_median = max { 300kB, median_100000blocks_longterm_weights }

The definition longterm_block_weight is

min { block_weight, 1.4 ∗ previous_effective_longterm_median }

The value of effective_longterm_median is the median of the last 100,000 blocks weights clamped between 300kB and 1.4 * previous effective_longterm_median. Same with cumulative_weights_median, it increases slowly and drops sharply. If all the blocks weights are greater than 1.4 ∗ previous_effective_longterm_median, effective_longterm_median rises by 40% every 50,000 blocks.

Bitcoin Core Fee Estimate Algorithm

me@iany.me (Ian Yang) — Sun, 16 Aug 2020 14:38:10 +0800

In bitcoin, the total size of transactions added to the chain in a specific time is limited. This creates a fee market. Transactions with a higher fee rate are likely to be confirmed more quickly. A good fee estimator predicates which fee rate to pay where there is a high probability the transaction will be confirmed into the chain within the target period.

Bitcoin core has the builtin support to estimate the fee rate. Understanding its algorithm can help us to migrate it and improve it.

This first part of this article introduces the core estimation algorithm. The algorithm determines which statistics data to track, and the second part shows how Bitcoin Core tracks them. The last part describes the extensions to the core algorithm for better performance.

The main reference of the algorithm is the source code and the post from the author.

The Core Estimation Algorithm

The algorithm must predicate which fee rate to pay where there is a high probability the transaction will be confirmed into the chain within the target period.

The basic idea is checking whether there is enough fraction of transactions with similar fee rate which has been confirmed within the target period.

Fee Rate Bucket

Bitcoin groups transactions into different buckets by fee rate to provide statistics for transactions with a similar fee rate.

The buckets are distributed exponentially. The buckets are configured via three parameters:

U: the first bucket upper bound.
S: spacing. The latter bucket upper bound is S times of the former bucket upper bound.
B: the total number of buckets.

The B buckets are separated by B - 1 upper bounds, which split the fee rate range from 0 to infinity into B segments, each segment corresponds to a bucket. The first upper bound is U, and the other bound is S times of its preceding bound.

The following is an example of 4 buckets.

Fee Rate Buckets

Confirmation Fraction

Bitcoin tracks the number three kinds of transactions in each bucket to compute the confirmation fraction.

Unconfirmed: the transactions remained in the memory pool. The lifespan of an unconfirmed transaction is the current chain height minus the chain height when the transaction was added to the pool.
Confirmed: the transactions included in the chain. The lifespan of a confirmed transaction is the number of the block containing the transaction minus the chain height when the transaction was added to the pool.
Failed: the transactions removed from the memory pool but not included in the chain. The lifespan of a failed transaction is the chain height when it is removed minus the chain height when it was added.

Further splitting transactions lifespan by the target period X blocks gives the following categories:

U: Unconfirmed and the lifespan is greater than or equal to X blocks
C: Confirmed and the lifespan is less than or equal to X blocks
T: Total number of confirmed transactions
F: Failed and the lifespan is greater than X blocks.

The confirmation fraction is:

Confirmation Fraction = C / (U + T + F)

Confirmation Fraction

The estimator needs to find the bucket in which confirmation fraction is larger than a specific threshold.

Buckets Scan

With the tracking data, the estimation algorithm scans the buckets to find an optimal bucket in which confirmation fraction is larger than a specific threshold for the given confirmation target.

There are two modes to find the optimal bucket which fee rate meets the confirmation target.

Greater All Passed looks for the lowest fee rate that all higher values pass. It starts at the bucket with the highest fee rate and looks at successively smaller buckets until reaching failure. The best bucket is the last passed bucket in the scan.
Less All Failed looks for the highest fee rate that all lower values fail. It starts at the bucket with the lowest fee rate and looks at successively larger buckets until passing the threshold. The best bucket is the last failed bucket in the scan.

Not every bucket has enough data points, Bitcoin will merge consecutive buckets until there are enough transactions, a.k.a, the total number of confirmed transactions are larger than a specific threshold. At the end of the scan, if all the remaining blocks combined have no enough data points, the estimation fails.

Buckets Merge

If the found best bucket is a merged bucket range, the one containing the transaction with the median fee rate is selected. And the estimated fee rate is the average fee rate of all the confirmed transactions in the bucket.

Median Bucket

Tracking

From the analysis in the previous chapter, given a specific bucket, the estimator has to use the following statistic metrics:

U: Unconfirmed and the lifespan is greater than or equal to X blocks
C: Confirmed and the lifespan is less than or equal to X blocks
T: Total number of confirmed transactions
F: Failed and the lifespan is greater than X blocks.
R: Sum of fee rates of all the confirmed transactions.

C and R are required to compute the average fee rate in the best bucket which equals to R / C.

For performance issues, Bitcoin tracks up to N blocks and uses a scale S to aggregate data. Pay attention that N and S will be used in the counters defined in the following chapters.

The tracking is performed on each chain height once thus ignores the fork chain which has the same height as a tracked chain.

The estimator needs to monitor following three kinds of events to track the statistics:

A new block is added to the chain.
A new transaction is added to the pool.
A transaction is removed from pool and the reason is not that it has been added to the chain.

Unconfirmed Transactions

Bitcoin uses a data structure to track the number of unconfirmed transactions for each lifespan between 0 (inclusively) and N (exclusively) and the number of all unconfirmed transactions which lifespan is greater or equal to N. It can give the answer to U: Unconfirmed and the lifespan is greater than or equal to X blocks, when X is not greater than N.

To avoid moving data around, the first N counters 0 to N-1 are organized as a ring. Assume that the current chain height is H, the transactions added at the height H - i belongs to counter r where r is the remainder after dividing H - i by N.

Tracking Unconfirmed Transactions

Rotating Unconfirmed Transactions

The counter N, the last counter, records the number of unconfirmed transactions which lifespan is greater than or equal to N. When a counter in the ring is about to be replaced, the count is added to the counter N.

See TxConfirmStats::unconfTxs and TxConfirmStats::oldUnconfTxs in the source code.

Failed Transactions

There are N / S counters, 0 to N/S-1, for each fee rate bucket to track failed transactions number. The counter i tracks the removed transactions in which lifespan is greater than i * S + 1. The counter ceil(X / S) - 1 gives an approximate answer to F: Failed and the lifespan is greater than X blocks.

Tracking Failed Transactions

The counters are keeping moving average instead of the total count, which means at the beginning of each new chain height, the old counter decays with a configured percentage decay which is less than 1.

count = count * decay

Decaying Moving Average

See TxConfirmStats::failAvg in the source code.

Confirmed Transactions

There are N / S counters, 0 to N/S-1, for each fee rate bucket to track confirmed transactions number. The counter i tracks the confirmed transactions which lifespan is less than or equal to (i + 1) * S. The counter ceil(X / S) - 1 gives an approximate answer to C: Confirmed and the lifespan is less than or equal to X blocks

Tracking Confirmed Transactions

There are two extra counters, one tracks the total number of confirmed transactions and another tracks the total fee rates of all confirmed transactions. They provide answers to T: Total number of confirmed transactions, and R: Sum of fee rates of all the confirmed transactions.

All the N / S and two extra counters for confirmed transactions, like the failed transactions are keeping the decaying moving average with the same decay percentage.

See TxConfirmStats::confAvg, TxConfirmStats::txCtAvg and TxConfirmStats::avg in the source code.

Extensions

Bitcoin use three data sets with different scales and maximum tracked confirmations.

Short: The scale is 1. It tracks 12 blocks and decays the moving average 96.2% on each new height.
Med: The scale is 2. It tracks 48 blocks and decays the moving average 99.52%.
Long: The scale is 24. It tracks 1008 blocks and decays the moving average 99.931%.

Longer time horizon can estimate the fee rate with a high confirmation target but is less accurate for a low target.

Given a required confirmation target X, bitcoin chooses the data set with the shortest tracking confirmations which are greater than or equal to X. For example, it uses Short when X is 12, and Med when X is 13.

The first extension, named as estimateCombinedFee, is that Bitcoin also checks the shorter time horizons and gives the lowest estimation as the answer. For example, when X is 49, the estimator will run the core algorithm three times:

Uses data set Long with target 49
Uses data set Med with target 48
Uses data set Short with target 12.

The second extension is that Bitcoin will run estimateCombinedFee with 3 different configurations and return the largest estimation given the confirmation target X.

Half Estimation runs estimateCombinedFee with confirmation target X/2 and confirmation fraction threshold 60%.
Actual Estimation runs estimateCombinedFee with confirmation target X and confirmation fraction threshold 85%.
Double Estimation runs estimateCombinedFee with confirmation target 2X and confirmation fraction threshold 95%.

Yubico for Windows

me@iany.me (Ian Yang) — Sat, 11 Jul 2020 16:38:25 +0800

This post records how I set up Yubico Key in Windows, so I’ll not delve into too much details. I have the model YubiKey 5 NFC. I frequently use 2 GPG keys stored in the key, one for encryption, another for SSH authentication.

The GPG encryption part is simple, GnuPG just works. Using the stored GPG key for SSH is a bit complex, because it requires collaboration between GnuPG and the SSH client. ~~After experiment many different solutions, I decide to use the simplest one, using putty/plink as the SSH client and enabling thepageant support in GnuPG.~~ See ♯ SSH Authentication Using a YubiKey on Windows And the OpenSSH Client how to use OpenSSH client with YubiKey.

Windows

I use scoop to manage apps in Windows. My configuration requires gpg and putty, where putty is in the extra buckets.

scoop bucket add extras
scoop install gpg putty

Tells git to use gpg and putty

git config --global gpg.program (Get-Command -Name 'gpg.exe').Source
$SSHPath = (Get-Command -Name 'plink.exe').Source
[Environment]::SetEnvironmentVariable('GIT_SSH', $SSHPath, 'User')

Edit gpg-agent.conf in folder $(scoop prefix gpg)\home\, enable putty support

enable-putty-support

By the way, the putty support means that gpg agent will also act as pageant, so there’s no need and it is also forbidden to run pageant manually. If pageant is running, quit it first then restart gpg agent.

Since my GPG keys are already stored in the YubiKey, I just need to export the public keys from somewhere and import them into the Windows host.

Restart gpg agent to check whether the keys are recognized:

gpg-connect-agent killagent /bye
gpg --card-status
gpg -K

If gpg --card-status complains that it cannot find the key card, refer to this article to save reader-port Yubico Yubi in the file ``$(scoop prefix gpg)\home\scdaemon.conf` and try again.

WSL

My solution to use the YubiKey in WSL is straightforward. Just use the Windows executables in WSL.

ln -snf "$(which plink.exe)" "$HOME/bin/ssh"
ln -snf "$(which pscp.exe)" "$HOME/bin/scp"
ln -snf "$(which gpg.exe)" "$HOME/bin/gpg"
git config --global core.sshCommand "$(which plink.exe)"

Memory Tracing in Windows

me@iany.me (Ian Yang) — Sat, 27 Jun 2020 16:52:48 +0800

The article mainly refers Exercise 2 - Track User Mode Process Allocations | Microsoft Docs.

1. Install Windows Performance Recorder and Analyzer

Download and install ADK from here. The windows build version can be checked by running “winver” via Win+r.

It’s sufficient to only install the component “Windows Performance Toolkit”

2. Enable Tracing the Target Process

For example, to enable tracking ckb

reg add "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\ckb.exe" /v TracingFlags /t REG_DWORD /d 1 /f

Pay attention to the executable name “ckb.exe” used in the command. The command requires administrator permission. A simple way is starting an admin PowerShell using keyboard shortcuts Win+x a.

3. Build the Process With Debug Symbols

In Rust, this can be enabled by adding debug = true to Cargo.toml

[profile.release]
debug = true

Now cargo build --release will also creates a pdb file, which contains the debug symbols.

4. Start Recording

Start Windows Performance Recorder from Start Menu. Check “Heap usage” and “VirtualAlloc usage” in the resource analysis. Then click the “Start” button.

5. Launch Process

Launch the process for a while, like 3 minutes. On my 16G memory machine, Windows Performance Analyzer seems cannot load too long recording. It will complain there is no enough memory.

6. Stop Recording and Save the Report

Reactivate Windows Performance Recorder and click the “Save” button.

7. Open the Report in Windows Performance Analyzer

Open the saved report file in Windows Performance Analyzer.

Expand the Memory in the left side bar and drag “VirtualAlloc Commit LifeTimes” chart to the Analysis tab.

VirtualAlloc Commit LifeTimes tracking records all the heap memory allocation, when they are allocated and freed. It can help us to find out who has allocated memories but not return them.

First filter the the result to only show the target process. Find the process in the table and apply “Filter to Selection” in the right click context menu.

The key columns of interest are the following (source):

Column	Description
Process	The name of the process that performs memory allocations through VirtualAlloc.
Commit Stack	The call stack that shows the code path leading to memory being allocated.
Commit Time	The timestamp of when memory was allocated.
Decommit Time	The timestamp of when memory was freed.
Impacting Size	The size of outstanding allocations or the size difference between the start and end of the selected time interval. This size adjusts based on the selected view port.The Impacting Size value will be zero if all memory allocated by a process is freed by the end of the visualized interval in WPA.
Size	The cumulative sum of all allocation during the selected time interval.

New columns can be added by right click the existing column header. The lines are automatically grouped by the columns that on the left of the yellow vertical bar.

However, I find out that it is hard to locate the leaf nodes when dragging the Commit Stack column to the left.

Sometimes, it is useful to zoom into a specfic time range, such as a memory surge to see the impacting allocations.

The Commit Stack does not loads debug symbols by default. First configure the symbols paths to include target process pdb file via menu item “Trace ➤ Configure Symbols Paths”.

Windows Performance Recorder also automatically saves system libraries debug symbols in a “NGENPDB” directory. Windows Performance Analyzer will automatically add it. But if the recording file is too large, and the computer is not powerful enought, the computer may stuck at loading symbols. If so, uncheck the “NGENPDB” in the symbols paths.

After setting up the paths, select menu “Trace ➤ Load Symbols”.

Now Commit Stack will show the symbols.

Move WSL to Another Drive

me@iany.me (Ian Yang) — Fri, 26 Jun 2020 16:51:34 +0800

Following example moves the Ubuntu distribution to disk D:\WSL\Ubuntu.

Step 1. Install Ubuntu in the Microsoft Store. Launch it to initialize the default instance. Create the user used in Ubuntu as prompted.

Step 2. Export the instance and import into the target directory.

cd D:\
mkdir WSL
cd WSL
mkdir Ubuntu
wsl --export Ubuntu .\Ubuntu\ext4.vhdx --vhd
wsl --unregister Ubuntu
wsl --import-in-place Ubuntu .\Ubuntu\ext4.vhdx

Hint

This is a faster version credited to Slawek. The old version has been moved to the chapter Alternative Step 2 for WSL1.

The commands above also unregister the default instance.

Command explanation:

wsl --export Ubuntu .\Ubuntu\ext4.vhdx --vhd: Export the disk of the WSL instance with name Ubuntu into the file ext4.vhdx in the directory .\Ubuntu\. So you will get the file D:\WSL\Ubuntu\ext4.vhdx.
wsl --unregister Ubuntu: Unregister the WSL instance with name Ubuntu.
wsl --import-in-place Ubuntu .\Ubuntu\ext4.vhdx: The first Ubuntu is the new created instance name. This command will register a new instance using the disk file .\Ubuntu\ext4.vhdx in place. Keep in mind that if the disk file is deleted, the instance will crash and all the files within it will be permanently lost.

Now it is also OK to uninstall Ubuntu in the store.

Step 3. Set the default user for the moved Ubuntu.

In Step 1, you have created a user for Ubuntu. After export and import, the new instance will use root by default. If you want to continue to use that user, please configure it via registry table.

Find the directory in registry HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss which DistributionName is “Ubuntu”. Set its DefaultUid to decimal 1000 (or hex 3e8).

Or create a file /etc/wsl.conf in the WSL instance with following contents:

[user]
default = yournamehere

Step 4. Migrate to WSL2

Use wsl -l -v to check whether the new created instance use WSL2. The command wsl --set-version Ubuntu 2 upgrades an instance with name Ubuntu to version 2.

Step 5. Try wsl

wsl -d Ubuntu

An alternative solution is using the tool LxRunOffline.

Alternative Step 2 for WSL1

Info

This is the old version of step 2, which is slower but also works for WSL1.

Step 2. Export the instance and import into the target directory.

cd D:\
mkdir WSL
cd WSL
wsl --export Ubuntu ubuntu.tar
wsl --unregister Ubuntu
mkdir Ubuntu
wsl --import Ubuntu Ubuntu ubuntu.tar

The commands above also unregister the default instance.

Command explanation:

wsl --export Ubuntu ubuntu.tar: Export the WSL instance with name Ubuntu into the file ubuntu.tar in the current directory. So you will get the file D:\WSL\ubuntu.tar.
wsl --unregister Ubuntu: Unregister the WSL instance with name Ubuntu.
wsl --import Ubuntu Ubuntu ubuntu.tar: The first Ubuntu is the new created instance name. The second Ubuntu is the instance saved location. The last parameter is the file created by wsl --export. This will import ubuntu.tar and use D:\WSL\Ubuntu to save the WSL instance data.

Now it is also OK to uninstall Ubuntu in the store and continue to the Step 3.

Podman on Windows via WSL2

me@iany.me (Ian Yang) — Tue, 23 Jun 2020 16:43:25 +0800

I prefer using a dedicated WSL instance to run containers. So I’ll install a minimal distribution, Alpine, to run podman.

Install the tool LxRunOffline first via scoop

scoop bucket add extras
scoop install lxrunoffline

Download Alpine root package from https://lxrunoffline.apphb.com/download/Alpine. See more distributions in LxRunOffline Wiki.

Then create a WSL instance from the package

lxrunoffline.exe i -n Alpine -f alpine-minirootfs-3.12.0-x86_64.tar.gz -d "D:\WSL\Alpine" -v 2 -r .

This will install Alpine into folder D:\WSL\Alpine. Although -v 2 is specified, but when I’m running it, the installed instance is still v1. It’s easy to fix by migrating to version 2.

wsl --set-version Alpine 2

Now start the instance

wsl ~ -d Alpine

Edit /etc/apk/repositories to switch to edge and add testing repository.

http://dl-cdn.alpinelinux.org/alpine/edge/main
http://dl-cdn.alpinelinux.org/alpine/edge/community
http://dl-cdn.alpinelinux.org/alpine/edge/testing

Install podman

apk update
apk add podman

It’s time to run Alpine inside Alpine

podman run --rm -it docker.io/alpine ash

Minimize any window into system tray in Windows

me@iany.me (Ian Yang) — Sat, 20 Jun 2020 16:39:57 +0800

There are two apps both work for me in scoop extras bucket.

scoop bucket add extras
scoop install rbtray traymond

They both require running their background process first. Rbtray minimizes a window into system tray by right clicking the window minimization button, while traymond us shortcuts Win + Shift + Z。

Rbtray has no its own system tray icon. Minimize a window and right click it to quit rbtray. Traymond has its own tray icon, use it to exit or restore all windows.

Vim Setup for Windows

me@iany.me (Ian Yang) — Tue, 05 May 2020 10:28:26 +0800

I used to use Visual Studio Code in Windows as mentioned in a previous post, ♯ My Windows Environment Setup. But its startup time is terrible on Surface Go, so I decide to give vim another try.

First install vim via scoop

scoop install vim

The most important difference of the Windows vim is that it uses different paths:

.vimrc -> _vimrc
.gvimrc -> _gvimrc
.vim -> vimfiles

For example plug.vim has to be saved as ~/vimfiles/autoload/plug.vim.

Following are some Windows specific options:

" Force color and encoding. Put these near the top of the config file.
if has("win32")
set t_Co=256
set encoding=utf-8
endif
if has("win32")
" Support different cursor shapes in Windows Terminal
let &t_SI="\<CSI>5 q"
let &t_EI="\<CSI>1 q"
" Disable fzf preview because it is broken in PowerShell
let g:fzf_preview_window = ''
" Use PowerShell as the shell
set shell=powershell.exe
set shellcmdflag=-NoLogo\ -NoProfile\ -NonInteractive\ -command
endif

And options in _gvimrc for gvim:

set guifont=JetBrains_Mono:h11
set guioptions-=m
" Show ligatures. It is not perfect and requires C-L to manually refresh.
set renderoptions=type:directx
" Fix the ugly cursor color
hi Cursor guibg=#005f87 guifg=#eeeeee

Pass, A Password Manager Utilizing GPG and Git

me@iany.me (Ian Yang) — Sun, 03 May 2020 22:50:53 +0800

A friend recommended gopass to manage passwords. After the trial, I decided to switch.

Gopass is indeed an implementation which follows the protocols defined by pass. It utilizes gpg to encrypt files and git to synchronize. It is really fast to manage the password vault because I already used to the command line environment and text editing tools.

I also use browserpass in Chrome and “Pass - Password Manager” in iOS.

I migrated from Enpass. There is a tool pass-import, but it does not work well on my exported Enpass vault, so I decide to build the pass store from scratch. Gopass indeed can encrypt and save any file, thus I just add the whole exported Enpass CSV file into the repository.

gopass edit enpass.csv

Copy and paste the exported Enpass CSV file into the opened editor and save. It is easy to search in the file using a pager or grep.

gopass show enpass.csv | less
gopass show enpass.csv | grep google

Now I can rebuild the password database with a create-on-use strategy: Whenever I want to use a password which has no its own entry yet, I search it in enpass.csv and create one.

My Windows Environment Setup

me@iany.me (Ian Yang) — Sun, 03 May 2020 22:03:22 +0800

I have only one Windows device, the Surface Go. I work on it occasionally, especially on short trips. I prefer Surface Go because of handwriting. I have a simple setup to meet my work requirements.

I spend the most time on a computer in three apps: terminal, text editor and web browser. I use Windows Terminal, Visual Studio Code and Chrome in Windows.

I use ColorTool to convert my favorite theme PaperColor into Windows Terminal schema. This is my full Windows Terminal config file.

Since I don’t do heavy development locally, I use PowerShell as default in the Windows Terminal. The command echo $profile will print the PowerShell config file path. The essential PowerShell config is enabling the Emacs style keybinding (full version here).

Set-PSReadLineOption -EditMode emacs

Scoop is very handy to install essential command line utilities, for example:

scoop install mingit ripgrep

I use the OpenSSH Client component in Windows, which can be enabled in “Settings / Apps / Manage optional features”. I used to set an SSH passphrase, but the ssh agent service is disabled. It can be enabled in PowerShell ran as admin:

# Run once as admin
Get-Service ssh-agent | Set-Service -StartupType manual

I have set the startup type to manual, so if I want to load the key, I have to start the service first:

Start-Service ssh-agent
ssh-add

However, the git installed by scoop uses its own bundled ssh client by default. It is easy to fix it by setting an environment variable GIT_SSH.

# Run once as logged in user
$SSHPath = (Get-Command -Name 'ssh.exe').Source
[Environment]::SetEnvironmentVariable('GIT_SSH', $SSHPath, 'User')

Restart Windows Terminal and now git will use the key in ssh agent.

Git has the same issue with GPG. Use git config to tell where the GPG program is:

git config --global gpg.program (Get-Command -Name 'gpg.exe').Source

Following is a list of other apps I used in Windows:

XMind: Mindmap tool.
Drawboard PDF: PDF reading and annotation.
Leonardo: an infinite canvas painting app.
Nebo: Handwriting notebook.
Clash for Windows
OneDrive and Google Keep for quickly synchronize data with other devices.

Write to Any iOS Files Location via Working Copy

me@iany.me (Ian Yang) — Fri, 24 Apr 2020 21:14:17 +0800

I use file system to manage my knowledge base. The repository has a well defined directory structure. I have a bunch of scripts to help me creating these directories on the laptop. But in iOS, I have to create them manually. Recently, I have found out a way to work around it via Working Copy.

The trick uses two features of Working Copy:

2-way synchronized repository with any iOS Files location.
The shortcut Write Repository File.

The shortcut allows writing file into any path in a repository, and the synchronized repository synchronizes changes back to the target iOS Files location.

Write to Files via Working Copy

In the following I’ll show how to create a shortcut which saves files into an iCloud folder and sorts them into sub-directories by date.

First create the target directory and set up the synchronized repository. I use Desktop/Inbox.

Set Up Sync Repo in Working Copy

Then create the shortcut. It accepts the file from Share Sheet and saves it into the synchronized repository sorted by date. It is important to check the option “Create” in the Write Repository File step.

Shortcut: Save Files by Date

Now give it a try. Share a text file using the new created action.

Test Shortcut

And the file appears in the Desktop/Inbox.

File Sorted into Inbox

This solution is not perfect and has two major drawbacks:

It duplicates the files.
It requires opening Working Copy to synchronize files.

However, it also has a bonus. I can backup the directory by git now.

Rust Pin

me@iany.me (Ian Yang) — Sat, 11 Apr 2020 21:08:33 +0800

Pin is an obscure type in Rust because of the naming and indirect concepts.

The first indirect concept is pointer. Pin<X> does not guarantee that X will not move. If X is a pointer which target type is T, Pin<X> guarantees that T will not move.

The second is “not move”. It really means that the only way to get the mut reference to T is via unsafe interface.

The last is the Unpin mark trait. Pin forbids safe interface to get the mut reference to T only when T is !Unpin.

In simple words, Pin is a pointer wrapper. When a pointer is trapped inside Pin, and the pointee type is !Unpin, there’s no safe way to get a mut reference to the pointee.

The following diagram has listed what Pin provides and constraints on when we can use those functions.

In Rust, the pointer is indeed the trait Deref and DerefMut, from which we can get the shared or mut reference.

It is not very interesting when the pointer is Deref. The essential of Pin is when X is DerefMut. The yellow box shows that the safe interface to get the mut reference is available when T is Unpin.

Unpin is implemented for types by default. It acts as a safety which disables the core feature of Pin. Pin is only effective when the safety is turned off, a.k.a, when the type is explicitly marked as !Unpin via PhantomPinned.

Pin is a signal that a self reference type may appear here. It is also a contract. The type provider must mark the type as !Unpin if it is unsafe to move. The type user must promise not to move the !Unpin data in the unsafe block.

Fix iOS App Store Family Purchase Sharing

me@iany.me (Ian Yang) — Sun, 22 Mar 2020 19:01:28 +0800

I have created a family group using the Apple ID A, and the account B is a member. However, I can’t download apps purchased by A in the device logged in by B. Today, I finally find out the cause. The Purchase Sharing setting in account B is incorrect, which is by accident set to share as A and I don’t know why.

To fix it, just ensure every family member has set to share purchase as themselves in the Purchase Sharing setting.

Ensure Correct Purchase Sharing Settings in the Family Member Accounts

In case you don’t know how to download the Family Shared Purchase:

Download Family Purchase Sharing

Save Web Articles via Inoreader

me@iany.me (Ian Yang) — Sat, 14 Mar 2020 17:02:25 +0800

One of my favorite features of Inoreader is saving the article to external services, such as Evernote, OneDrive or Google Drive. I can archive the articles I liked for future references.

Inoreader even supports automatically fetching full text for truncated RSS feed. This is awesome to work with other web services. Many bookmarking and reading later services, such as Diigo, Instapaper and Pinboard, all provide RSS feed but without full text.

Automatically Fetch Full Content

The Inoreader Pro subscribers can save the articles automatically by creating rules. For example, I have saved all of my bookmarks into Evernote.

Automatically Save Articles

Saved Articles

Blocking Stdout

me@iany.me (Ian Yang) — Sun, 01 Mar 2020 09:29:40 +0800

When I first read Stjepan’s article Blocking inside async code, I never though I will met the problem mentioned in the post.

… I bet we all most of the time assume printing to standard output does not block while it really could.

In case you’re wondering why println!() can block, imagine we executed program1 | program2 in a shell so that the output of program1 is piped into program2. If program2 is reading input very slowly, then program1 will have to block whenever it prints something and the pipe is full.

Fortunately, my brain has stored the clue somewhere, and I can retrieve it and save my day when I heart a weird bug.

We have a GUI app N written in Node which bundles a service binary C. N starts C as a sub-process. C writes logs to a file. After 10 minutes, it stops writing the log file until restarted.

C uses a single thread to write logs. But default, it sends the logs to both stdout and log file. When the stdout buffer of C is full, the logging thread stuck, so it also stops writing the log file.

A simple fix is telling C not write to stdout. But the root cause is that the app N keeps the child process stdout pipe open and never read from the pipe. The best practice is that if you don’t read from the pipe, close it.

Following is an example In Node to close both stdin and stdout via option stdio. The stderr is open because the parent process will read from the pipe.

const c = spawn('c', [], {
stdio: ['ignore', 'ignore', 'pipe']
});
c.stderr.on('data', (data) => {
console.log(`c: ${data}`);
});

Excel as Diagram Maker

me@iany.me (Ian Yang) — Sat, 22 Feb 2020 21:18:00 +0800

Excel is a full featured vector graphics app. It has bundled many shapes and styles, and even supports anchoring connectors to the shapes. All these features make Excel also a good diagram maker.

One of my favorite tips is grid snapping, it can save a lot of time to align shapes. I also recommend resizing the grid as small squares.

I have recorded a screencast for it. The video has demonstrated:

Resize the grid into small squares.
Insert a shape to open the Shape Format menu.
Enable snapping features in the Align menu.
Insert more shapes and add connectors.

Draw Diagrams in Excel

Bitcoin Core Network Event Loops

me@iany.me (Ian Yang) — Wed, 12 Feb 2020 20:22:00 +0800

This article is an analysis of the network event loops based on bitcoin core v0.19.0.

Bitcoin starts two threads to handle network messages, and each thread runs its own event loop.

Socket Handler Thread

The Socket Handler Thread is responsible for the underlying socket IO.

It loops each connected peer in a round-robin schedule. The handler reads messages from the peer socket into the receiving queue vProcessMsg, and sends message in the sending queue vSendMsg to peer.

Socket Handler

Read from Socket

First, the thread tries to read some data from current peer.

Read some data from socket.
Decode the message. The newly decoded messages are stored in a queue named vRecvMsg.
Move the decoded message in vRecvMsg to another queue vProcessMsg. The queue vProcessMsg is the inbox for the Message Handler Thread.
If there is any new decoded message in this loop step, try to wake the Message Handler Thread.

Write to Socket

Then, the thread sends queued messages in vSendMsg to current peer.

Code References

CConnman::Start: This is the main entry to start network threads.
CConnman::SocketHandler: The main logic of Socket Handler Thread.
CNode::ReceiveMsgBytes: Decode received bytes into message.

Message Handler Thread

Message Handler Thread handles the network messages without worrying the socket operations.

The behavior of the Message Handler Thread is similar to the Socket Handler Thread. Both loops each connected peer in the round-robin way. And for each peer, they first process incoming messages, and then send outgoing messages. The difference is that Message Handler Thread reads messages from the queue vProcessMsg and queues outgoing messages in the queue vSendMsg.

Message Handler

The two loops are connected at these two message queues.

Two Loops are connected on two queues.

Process Incoming Messages

The Message Handler Thread processes the incoming messages one by one in the function ProcessMessage with a huge switch structure.

Each peer acts as a state machine. For some reasons, the state is stored in two different locations.

The CNode structure, which is passed around as a function parameter, such as pfrom in ProcessMessage.
The CNodeState stored in a global table.

Schedule more Outgoing Messages

Besides the queued messages acting as responses to the incoming messages, the Message Handler Thread also schedules more outgoing messages.

There are mainly two kinds of outgoing messages in this step:

Messages depending on timer, either periodic or the follow ups upon time out.
Messages queued and should be sent in batch, such as blocks and transactions announcements.

Code References

CConnman::ThreadMessageHandler: The Message Handler Thread.
::ProcessMessage: This function processes a single incoming message.
PeerLogicValidation::SendMessages: Check timer and batch queue and schedule more outgoing messages.

Vim iTerm Launcher

me@iany.me (Ian Yang) — Sat, 08 Feb 2020 16:48:30 +0800

I prefer using Vim in a terminal. I can switch to the shell with Ctrl-Z and back with fg. However it is hard to integrate a terminal command with other GUI tools, such as editing a file in Vim from Finder.

In macOS, the default Terminal app and iTerm both supports automation. It’s easy to write a script to open a new terminal window and run a command like “vim file” in it. But I want to close the window after Vim quits. A quick work around is running the following command:

vim file; exit 0

Now the problem is that if I suspend Vim via Ctrl-Z, the terminal window is closed, because shell will continue to execute the next command when the process is suspended.

After research and reading a StackOverflow answer, I wrote two scripts to launch Vim in iTerm. I also added features in iterm-vim-wrapper to edit clipboard and empty scratch file in temporary directory.

This is an example which uses the scripts to create File Action in Alfred.

How to Download Only Gmail Inbox

me@iany.me (Ian Yang) — Sun, 02 Feb 2020 12:15:10 +0800

I prefer reading my mails in the Gmail web client directly. I rarely send new mails or replies. But when I do, I want to use PGP to encrypt or sign the mail. I have tried two extensions to use PGP in Gmail, FlowCrypt and Mailvelope. But both of them are far from a competent solution for me. GPG Mail in the GPG Suite for macOS is still my preferred way. However, GPG Mail is an Apple Mail plugin, which requires downloading mails first. Based on my scenarios, I only need to download the mails I left in the inbox. But the basic setup of Gmail in Apple Mail will download many mails in the background.

I tried POP3. It has two issues:

The status is not synchronized with the server. I can configure Gmail to archive the mails which have been downloaded via POP3, but that means once I have downloaded mails, I must process them locally. This is a burden, since I only want to reply several mails in the client.
POP3 does not observe the filter which auto archives new mails. The filtered emails still occur in the client.

IMAP is still the better choice. The real issue is indeed that Apple Mail uses remote IMAP mail boxes for Drafts, Sent, Junk, Trash and Archive by default. If it uses local mail boxes instead, the client will no longer try to synchronize this mail boxes.

Following is the detailed instructions to setup a Gmail IMAP client which only downloads mails in the inbox.

First go to the Gmail label settings. Disable all the “Show in IMAP” options except the one for inbox.

Since Apple Mail does not allow configuring mailboxes for a Google account, I have to generate an app password to add the Gmail as an IMAP account. Open Mail app, add account and choose IMAP from “Other mail account…”.

After adding the new account, use local mail boxes in the Mailbox Behaviors settings pane. Leave the Archive mail box remote, since we cannot change it.

Apple Mail Mailboxes Behaviors Settings Pane

Now Apple Mail will only download mails in the inbox. There is one last thing, if I archive a mail in the client, it will be labeled [Imap]/Archive in Gmail, you may want to hide it from the label list and message like me in the labels settings.

Allocate Energy to Categories

me@iany.me (Ian Yang) — Sun, 18 Aug 2019 12:42:26 +0000

I have read the post Get Everything Done & Still Have Time to Play by Jackie Ashton recently and adopted some strategies into my process.

Before I plan the tasks daily by allocating them into time slots in a calendar, it is tedious and time-wasting. It messes up my agenda, which is intended only to contain the events that I must do on time. And it makes me nervous and exhausted to follow a pre-defined schedule every day.

My new process is much simpler. I review my life objectives first and group them into categories. Then I evaluate their importance in my current life stage and allocate my energy in percentage into them.

Here is the list looks like now:

Maker 30%: Work as a programmer. I follow the suggestion from the lesson one “Switch Between Manager And Maker Modes” in the book “Leading Snowflakes.”
Manager 5%: Work as a manager.
Career 15%: The tasks that are not directly related to my work but help to improve myself on my career path.
English 15%: I want to improve my English listening and speaking, so I commit enough time on it. However, it is a short term objective. Once I’m satisfied with my English level, I can move the time to other categories.
Hobbits 10%: Piano, painting.
Workout 10%: Running, swimming.
Misc 5%: Such as reading novels.
Family 10%: Quality time with family.

Then I create a punch card to plan timed bursts for each category. I use 50 minutes for tasks requiring long focus period, and 25 minutes for short focus period. The 50 minutes session counts for 10% of the energy, and 25 minutes accounts for 5% because I dedicate 10 hours per day to the planned tasks. And this is my punch card.

Maker 50 / 50 / 50
Career 50 / 25
English 25 / 25 / 25
Hobbits 25 / 25
Workout 50
Manager 25
Misc 25
Family 50

I have set up my OmniFocus projects hierarchy by categories. Every night, I go through each category to see what to do next and plan enough tasks for the next day according to the allocated energy.

The next day, I choose a category and pick a task to work. If I have completed the timed burst, I checked it in the punch card, then move on to the next one.

The new process has many pros:

It is simple. Simplicity saves time and is efficient.
It associates every task to one of my life objectives. It motivates me to complete the job if its objective is crucial for me. Otherwise, if I always feel reluctant to do a task, maybe I should cross on that objective from my life temporarily.
I can dynamically adjust my day. If some routines occupy me in the most time of the day, I can use the remaining time to only the most critical categories.
It reminds me of what is essential in my life. For example, I should spare enough time with my family and friends.

The cons are that I should take the punch card with me. But it is so simple that I can print a whole week into a single piece of A5 paper. As a bonus, I can review the card every week, record the completion status into a spreadsheet, and adjust my plan according to the statistics.

How to Mock Time in Rust Tests and Cargo Gotchas We Met

me@iany.me (Ian Yang) — Sun, 31 Mar 2019 06:22:22 +0000

How to mock time in Rust tests and Cargo gotchas we met. I’m working in a team developing a big Rust project recently. The project has some features depending on time. We, the developers, want to be able to mock the time in the test. In this post, I’ll talk about the problems we have met, mostly related to Cargo.

I have created a GitHub repository doitian/rust-mock-time-demo, which contains all the following examples.

The First Attempt

The requirement looks straightforward at first glance since Rust supports conditional compilation and cfg test is only active in the test. We can implement a function telling us current time, and the whole program must fetch the current time from it. The function has two different versions, in the non-test code, it just returns the real current time. In the test, it is possible to mock the current time through a thread local variable.

/// cfg-test/src/main.rs
use std::thread;
use std::time::{Duration, SystemTime, SystemTimeError};
#[cfg(not(test))]
pub fn now() -> SystemTime {
SystemTime::now()
}
#[cfg(test)]
pub mod mock_time {
use super::*;
use std::cell::RefCell;
thread_local! {
static MOCK_TIME: RefCell<Option<SystemTime>> = RefCell::new(None);
}
pub fn now() -> SystemTime {
MOCK_TIME.with(|cell| {
cell.borrow()
.as_ref()
.cloned()
.unwrap_or_else(SystemTime::now)
})
}
pub fn set_mock_time(time: SystemTime) {
MOCK_TIME.with(|cell| *cell.borrow_mut() = Some(time));
}
pub fn clear_mock_time() {
MOCK_TIME.with(|cell| *cell.borrow_mut() = None);
}
}
#[cfg(test)]
pub use mock_time::now;

Let’s try it in both cargo run and cargo test:

cargo run -p cfg-test
cargo test -p cfg-test

Work as expected, let’s call it a day.

Gochas of cfg test

However, we immediately found the issue when we try to add the mockable now into the project. The project is complex and organized into many crates, so we create a crate for this utility as well. However, the compiler complains that it cannot find the function mock_time::set_mock_time when using the cfg(test) only functions in the test in another crate. Since Cargo builds each file under tests directory in a standalone crate, see how it can reproduce the issue, see cfg-test/tests/test_mock_time.rs reproduces this issue using the command below

RUSTFLAGS='--cfg cfg_test_crate_tests' cargo test -p cfg-test

The cause is that cfg(test) does not pass though dependencies. The crate test_mock_time in tests is built with cfg(test), but in its dependency, the crate cfg-test , cfg(test) is not set. In simple words, cfg(test) is only set for the crate current in test.

Feature

“Feature” is a well-known feature of Cargo, where a crate can customize how to build its dependencies. It is easy to switch to feature, just change cfg(test) to cfg(feature = "..."), for example:

use std::time::SystemTime;
#[cfg(not(feature = "mock-time"))]
pub fn now() -> SystemTime {
SystemTime::now()
}

The full example is in the cfg-feature-lib directory in the repository.

Since the feature does not turn on in test automatically, we have to remember to enable the feature when running test. What’s worse, the command line argument --features does not pass to workspace members, it is only for the top project. Take the repository as an example:

Running following command in the top directory does not enable feature mock-time in cfg-feature-test-manual crate.

cargo test --features mock-time -p cfg-feature-test-manual

Instead, it must be executed inside cfg-feature-test-manual:

cd cfg-feature-test-manual && cargo test --features mock-time

After research, a trick comes out, which adds the dependency in both dependencies and dev-dependencies. The feature is only enabled in dev-dependencies. See example cfg-feature-test-auto/Cargo.toml.

[dependencies]
cfg-feature-lib = { path = "../cfg-feature-lib" }
[dev-dependencies.cfg-feature-lib]
path = "../cfg-feature-lib"
features = ["mock-time"]

We expect that mock-time is automatically enabled in tests, but it turns out to be where weird things happen.

In simple words:

If the top workspace project has added the conditional dependencies in Cargo.toml, than the feature is always enabled, no matter in tests or final executables. See a demo in this PR.
If the top workspace does not have such trick, then cargo behaves differently when whether --all is specified or not. Another PR demonstrates this.

The feature based solution also has a inconvenient drawback. The feature can only be passed from a crate to its direct dependencies. To allow the time mock, a crate must be aware of whether its dependency depending on time directly or indirectly.

For example, if the crate time has a feature mock-time, and

package foo depends on bar, and bar depends on time.
package root depends on foo, bar and time.

Then here is what root’s Cargo.toml looks like:

[dependencies]
foo = "0.1.0"
bar = "0.1.0"
[features]
mock-time = ["foo/mock-time", "bar/mock-time", "time/mock-time"]

Final Solution

Because of the weird behavior of --all, and how disturbing to set up the feature chain between dependencies, we decide to adopt RUSTFLAGS. Indeed, I have used it once in an example above. RUSTFLAGS is automatically enabled for all crates, no matter how deep the dependency is.

However, there is another gotcha about RUSTFLAGS. The doc test does not observe RUSTFLAGS, it uses RUSTDOCFLAGS.

We have built a crate faketime from our experiences. Take a look if you are interested in mocking time in tests.

Rust Cell and RefCell

me@iany.me (Ian Yang) — Sun, 24 Feb 2019 07:35:47 +0000

In Rust document, Cell is “A mutable memory location”, and RefCell is “A mutable memory location with dynamically checked borrow rules”.

They both provide “interior mutability”, where you can modify the value stored in cell via immutable reference of the cell.

They both have an API get_mut to return a mutable reference to the underlying data. This method requires a mutable reference to the cell, which guarantees that the callee has exclusive ownership of the cell.

pub fn get_mut(&mut self) -> &mut T

The difference is how they implement interior mutability. Cell copies or moves contained value, while RefCell allows both mutable and immutable reference borrowing. I will try to explain the difference via their APIs in this article.

Cell

In the old version of Rust, Cell requires the wrapped type to be Copy. Many articles still contain such outdated and misleading information. Indeed Cell has two different sets of APIs in a newer version.

The first set is the Copy API. It requires a Copy wrapper, and contains methods get and set. The method get returns a copy of the contained value, and set stores a copy of the argument val as the new value.

// impl<T: Copy> Cell<T>
pub fn get(&self) -> T
// impl<T> Cell<T>
pub fn set(&self, val: T)

Another set is the Move API. It has two methods take and set. The method take moves out the contained value, leaving Default::default() in its place. The call set also works for non-copyable type, where it moves the argument into the cell.

// impl<T: Default> Cell<T>
pub fn take(&self) -> T
// impl<T> Cell<T>
pub fn set(&self, val: T)

The method replace is an alternative of take, when T does not implement Default or the new value is known in advance when taking the value.

// impl<T> Cell<T>
pub fn replace(&self, val: T) -> T

An important property of Cell is that the cell cannot tell you what’s contained in the cell via a reference. You either copy the contained value, or modify the cell and move out the value.

RefCell

RefCell allows borrowing immutable or mutable reference to the contained value. It tracks the borrows at runtime, via borrow and borrow_mut.

pub fn borrow(&self) -> Ref<T>
pub fn borrow_mut(&self) -> RefMut<T>

The method borrow grants temporary access to the contained value via immutable reference. Multiple immutable borrows can be taken out at the same time. It panics if the value is currently mutably borrowed.

The API borrow_mut mutably borrows the wrapped value. It panics if the value is currently borrowed, either mutably or immutably.

The runtime tracking certainly has overheads, and RefCell also can lead to runtime panics.

My Reading Workflow and IFTTT Applets

me@iany.me (Ian Yang) — Sat, 12 Jan 2019 09:41:28 +0000

I categorize reading into 3 different categories:

Skim Reading: This includes the feeds I subscribed in Inoreader, websites I frequently visit and articles shared from my friends. I quickly skim them using fragmented time, usually less than half an hour in total per day, and save the articles which worth reading to Instapaper.
Insensitive Reading: Instapaper sends me 20 unread articles to Kindle every morning. If I can spare at least 10 minutes, I will read them on the Kindle.
Challenging Reading: Papers are definitely in this category. There are also articles hard to understand, I will print them into PDF. I schedule the time in advance, usually one hour or more, and read the PDF files on Sony Digital Paper.

I’ll share the detailed workflow and the IFTTT applets used in the process.

Sometimes, I also save some articles in Medium, so I have an applet to synchronize them to Instapaper.

Medium saved article → Instapaper

I aggregate all the stuffs which I think worth to share into Buffer. There is a large bundle of IFTTT applets for this:

My new blog post → Buffer
Instapaper liked → Buffer
Inoreader starred → Buffer
Medium Applause → Buffer
Reddit → Buffer
Youtube liked → Buffer
Pinboard tagged tt → Buffer
500px → Buffer
Lofter → Buffer

Buffer schedules the time and will finally publish to Twitter.

Buffer also acts as my sharing hub. I forward the shared messages to two other places, Telegram channel and Inoreader saved web pages.

Following applets publish to my Telegram channel

Buffer → Telegram ianshare
Liked Tweets → Telegram ianshare
New journal post → Telegram ianshare

The similar group of applets save them into Inoreader.

Buffer → Inoreader tagged ianFeed
Liked tweets → Inoreader tagged ianFeed
New journal post → Inoreader tagged ianFeed

I tagged the saved articles, so Inoreader allows me to share it as an RSS feed or an HTML page.

I review these shared articles weekly, and include them in my the journal posts.

The last, I backup the data into Google Drive spreadsheets via these applets.

My tweets → Google Drive
Liked tweets → Google Drive
Pinboard → Google Drive
Buffer → Google Drive

Weekly Paper: Liberal Radicalism

me@iany.me (Ian Yang) — Sat, 01 Dec 2018 12:06:51 +0000

paper source

(LR is) as flexible and responsive as the market, but avoids free-rider problems.

Flexible and responsive: any one can propose a new public good project, and the project can get enough fundings even when only a small community funds it.
Subsidies create incentives for citizens to fund projects.

Background

(Free-rider problem is) due to the expense or inefficiency involved in excluding individuals from access.

Cons of Existing Solutions

1p1v: oppress minorities.
Capitalism: inefficiently exclude potential users.
Charitable organization: difficult to closely align reliably with the common good.
QV: it doesn’t solve the problem of flexibility, a.k.a., it requires a curated projects list.

Model

Assumptions

We can verifiably distinguish among and identify these citizens.
Any citizen may at any time propose a new public good.
Our interest here is in maximization of dollar-equivalent value rather than achieving an equitable distribution of value.
Utility function V is concave (国内的叫法一般是反的，即我们平常说的凸函数), smooth, increasing.
The deficit is not bounded, a.k.a, the funding solution can collect unbounded taxes.

Optimization Target Function

Given:

Let $V_i^p(F^p)$ be the currency-equivalent utility function. It is the benefit that the citizen $i$ can get when the project $p$ is funded with $F^p$ units of currency.
The $i \times p$ contributions matrix $\left\{c_i^p\right\}_i^p$ , where $c_i^p$ is the contribution citizen $i$ makes to project $p$ .

We need to find a funding distribution solution $\left\{F^p\right\}^p$ which is a p-dimention vector. $F^p$ is the funds allocated to project $p$ . The solution maximize:

 $\sum _ { p } \left(V _ { i } ^ { p } \left( F ^ { p } \right) - c _ { i } ^ { p }\right) - t _ { i }$

where

 $\sum _ { i } t _ { i } = \sum _ { p } \left( F ^ { p } - \sum _ { i } c _ { i } ^ { p } \right)$

Since V is concave, smooth and increasing, it is easy to find the maximum using the first order derivative, which gives

 $V ^ { p ^ { \prime } } = 1$

Capitalism

 $F ^ { p } = \sum _ { i } c _ { i } ^ { p }$

Result

 $V ^ { p ^ { \prime } } = N$

1p1v

 $N \cdot \operatorname{Median}_{i} V_{i} ^ { p ^ { \prime } } \left( F ^ { P } \right) = 1$

The optimal solution requires mean, where median is absolutely different with mean.

LR

 $F ^ { p } = \left( \sum _ { i } \sqrt { c _ { i } ^ { p } } \right) ^ { 2 }$

(We) assume that citizens ignore their impact on the budget and costs imposed by it.

After incorporating the deficit:

 $V ^ { p ^ { \prime } } \approx 1 + \Lambda$

It is assumed that $\Lambda$ is on the order of $1/N$ .

Extensions

Budgeted matching funds

CLR: linear combine LR and Capitalism until the deficit is under the budget.

 $F ^ { p } = \alpha \left( \sum _ { i } \sqrt { c _ { i } ^ { p } } \right) ^ { 2 } + ( 1 - \alpha ) \sum _ { i } c _ { i } ^ { p }$

LR allows every project gets the optimal funding by incentivize citizens via the deficit. CLR is more practically, because in real world, there is always a budget.

Collusion

Coercion Resistance: A voter cannot prove to anyone else who they voted for (or even, ideally, whether or not they voted) even if they wanted to.

Negative contributions

More broadly, negative contributions may be a quite powerful way to deter collusive schemes as they offer a way for any citizen to be a “vigilante enforcer” against fraud and abuse.

On the other-side, it also can be used to attack and threaten other communities.

Weekly Paper: 1945 - Hayek - The Use of Knowledge in Society

me@iany.me (Ian Yang) — Sun, 25 Nov 2018 06:15:15 +0000

Economic problem of society is mainly one of rapid adaptation to changes in the particular circumstances of time and place, it would seem to follow that the ultimate decisions must be left to the people who are familiar with these circumstances, who know directly of the relevant changes and of the resources immediately available to meet them.

The knowledge is dispersed and rapid changes. To take advantage of this knowledge, people must make decision distributively. People require only relevant information to make better decisions. This communicating mechanism should be

communicating to him such further information as he needs to fit his decisions into the whole pattern of changes of the larger economic system.

It is always a question of the relative importance of the particular things with which he is concerned, and the causes which alter their relative importance are of no interest.

The price system solved it, and

We must look at the price system as such a mechanism for communicating information.

It is an effective communicating system because:

Only the most essential information is passed on and passed on only to those concerned.

It extends the span of our utilization of resources without the control of any one mind. Also, it will make the individuals do the desirable things without anyone having to tell them what to do.

Thoughts

It opened my mind to look at the price system differently. It is an effective way to watch the only the necessary and relevant information, and the messages are propagated rapidly in the system. It makes distributed decisions possible and outperforms central decision authorities in most cases.

Although distributed decisions can achieve local optimum, which may not be the global optimum. We human may also need some centralization to break the equilibrium by a trial and error like method.

References

Weekly Paper: Helix

me@iany.me (Ian Yang) — Sat, 10 Nov 2018 03:49:28 +0000

Orbs - Read the Helix Consensus Algorithm White Paper

Hazel is a Byzantine fault-tolerant and scalable consensus algorithm for the fair ordering of transactions among nodes in a distributed network.

It assumed that:

Node-node connections are assumed to be strongly synchronous.
There is a known bound for the faulty nodes.

It is scalable because the PBFT committee size is bounded.

Helix archives fairness in these aspects.

Committee selection relies on reputation and a random seed.
Transactions are randomly selected when they are encrypted.

Both committee election and transactions sampling utilize a random seed derived from the previous decrypted block.

Helix nodes validate block transaction by checking the distribution overlap with local transaction pool.

Flatbuffers Compatible Table

me@iany.me (Ian Yang) — Fri, 02 Nov 2018 02:00:36 +0000

parent	table	struct	union	vector	string	enum	scalar
root	Y	Y
table	Y	Y	Y	Y	Y	Y	Y
struct		Y				Y	Y
union	Y	Y
vector	Y	Y	Y		Y	Y	Y

If there is a Y in row A column B, B can be used as a child of A. If it is blank, B is not allowed as a child of A.

Parent root means which type can be used as a root object.

• the test schema

Decred Review

me@iany.me (Ian Yang) — Thu, 19 Jul 2018 01:24:39 +0000

Official References

Based on Bitcoin
PoS overlays upon PoW
Add signature algorithms Ed25519 and Schnorr
Use BLAKE-256 as hash algorithm
Enable segwit to solve transaction malleability
Transaction expiration mechanism

Hybrid Consensus

Decredible | Decred hybrid consensus explained

PoW miners propose new block.
PoS miners vote whether accepting the block. The ticket should be purchased first. 5 tickets are selected randomly using the new PoW block header. Block is accepted if 3 tickets agree.
Block reward is split between PoW and PoS miners.
PoS tickets are selected using PoW block header. It guaranteed that is has a very high cost to manipulate picking tickets.

Vim Marks Stack

me@iany.me (Ian Yang) — Thu, 31 May 2018 03:30:42 +0000

I rarely used more than two marks in Vim, so why not use them as a stack and keep latest history?

The idea is that

First shift existing marks in a-y to b-z, thus original mark z is dropped.
Then save current position into mark a.

function! PushMark(is_global)
if a:is_global
let l:curr = char2nr('Z')
else
let l:curr = char2nr('z')
endif
let l:until = l:curr - 25
while l:curr > l:until
call setpos("'" . nr2char(l:curr), getpos("'" . nr2char(l:curr - 1)))
let l:curr -= 1
endwhile
call setpos("'" . nr2char(l:curr), getpos("."))
endfunction
" Push to marks a-z
nnoremap <silent> m, :call PushMark(0)<CR>
" Push to marks A-Z
nnoremap <silent> m. :call PushMark(1)<CR>

Read Various RSA Keys in Ruby

me@iany.me (Ian Yang) — Sat, 18 Nov 2017 10:46:07 +0800

I recently worked in a Ruby on Rails project which should integrate with many different payment systems. There system mostly use RSA in encryption and signature. However they provide the RSA keys in different formats, it is a challenge to choose a right way to read the keys in Ruby.

RSA is an asymmetric cryptographic algorithm, thus it requires two keys, private key and public key. The key itself is just binary, but it can be encoded in different format.

DER/PEM

DER is a binary encoding method. It is consisted of unreadable characters. The extension is usually .cer, .crt, or .der.

PEM encode key using base64, then add PEM header and footer around key. For example

This is the most frequently used format, since ssh-keygen generate private key in PEM format. The PEM key file extension is usually .pem.

Both DER and PEM format can be read in Ruby using following code:

k = OpenSSL::PKey::RSA.new(File.read("/path/to/keyfile"))

If the key file has a passphrase, it can be specified as the second argument.

The method k.private? can be used to check whether the key is a private key. Since public key can be calculated from private key, public key is always available.

Sometimes, the key is provided just in base64 format without PEM header. There are two solutions:

Base64 decode the string and pass it as key content

k = OpenSSL::PKey::RSA.new(Base64.decode64(key_content))

Format it using PEM and then read the key

pem = key_content.gsub("\r\n", "").scan(/.{1,64}/).join("\n")
# or use BEGIN PRIVATE KEY
pem = "-----BEGIN PUBLIC KEY-----\n#{pem}\n-----END PUBLIC KEY-----\n"
k = OpenSSL::PKey::RSA.new(pem)

PKCS12

PKCS12 key file extension is usually .p12 or .pfx. It is a frequently used format in browsers to export certificates. Most PKCS12 files are protected using password, which is the optional second argument in the constructor.

pkcs12 = OpenSSL::PKCS12.new(key_content, certificate_password)
k = pkcs12.key

X509 Certificate

X509 Certificate adds some meta information to key, such as issuer, expiration date. X509 can be encoded using DER or PEM, the key file extension is usually .cer. In PEM format, the header is BEGIN CERTIFICATE.

X509 Certificate can be read as following:

cert = OpenSSL::X509::Certificate.new(File.read("/path/to/certfile"))
k = cert.public_key

When asn1 error is thrown, try switch between OpenSSL::X509::Certificate and OpenSSL::PKey::RSA.

Nginx as UDP Load Balance

me@iany.me (Ian Yang) — Wed, 23 Aug 2017 23:11:35 +0800

Just upgraded Graylog to cluster in a project. Because syslog UDP is used as input, a UDP load balance is required to distribute logs to servers in the cluster. Since the servers are hosted in Aliyun, I tried Aliyun UDP Load Balance first. But it does not forward requests evenly, and health detection diagram cannot be disabled. The popular HTTP load balance tool HAProxy does not support UDP. Fortunately, Nginx can be used as a UDP load balance.

First, the latest version of Nginx is required to support stream directive. For example, the default package in Ubuntu 14.04 does not support it, thus use PPA or compile from source.

The PPA version requires loading the feature from dynamic library. Add following line in the beginning of nginx.conf.

load_module /usr/lib/nginx/modules/ngx_stream_module.so;

The following snippet configures Nginx listens on UDP 1515, and forward requests evenly to the 1514 UDP port in three different servers. The snippet must be at the top level of nginx.conf. Files in sites-enabled usually are embedded in http directive, do not put stream config there.

The option proxy_responses is the number of packets expected from backend servers in responded to client. It should be configured according to scenario. In my case, I use it to accept logs forwarded from rsyslog. The client does not expect response, and the server does not respond at all. If the option is not set to 0, Nginx will refuse new connection after used up all of them.

stream {
upstream syslog_udp {
server logs1-xy.example.com:1514;
server logs2-xy.example.com:1514;
server logs3-xy.example.com:1514;
}
server {
listen 1515 udp;
proxy_pass syslog_udp;
proxy_responses 0;
}
}

Because there are so many logs requests, the open file limit and worker connections must be set to a large value.

worker_rlimit_nofile 1000000;
events {
worker_connections 20000;
}

Redis as Write Buffer

me@iany.me (Ian Yang) — Sat, 19 Aug 2017 23:41:07 +0800

It is common to use Redis as read buffer。To read data, first check whether it exists in Redis. If so, use the cached data, otherwise read from the backend storage and save a copy into Redis. To write data, first save into backend storage, then clear or update Reids cache.

But if the system bottleneck is in writing, the solution above does not work. But it is easy to modify it into a write buffer.

Read: Check whether the data exists in Redis. Read from backend storage if not.
Write: Just write into Redis. Notify background worker via message queue to flush the cache into backend storage.
The background worker watches message queue, save data and delete from Redis.

The message queue can be implemented using Redis LIST. Official RPOPLPUSH – Redis command document already described how to implement a reliable queue. The remaining issue is how to safely delete saved data from Redis.

The save step can be split into:

Read from Redis.
Save into backend storage.
Delete from Redis.

If new change comes between 1 and 3, the change is discarded in step 3. It must be resolved using lock, or transaction.

After research, it is a bit complex to implement a lock. Fortunately, it is easy to use transaction.

Redis provides MULTI and EXEC to wrap several commands into a transaction. Although it does not support rollback on error, it guarantees that either none of the commands are executed, or all of them have been executed. Command WATCH can monitor the changes on a key. If the key changes after WATCH and before executing the transaction, the transaction is cancelled. See details in official document here. So the save step can be implemented in following steps:

WATCH the key to flush
Read from Redis.
Save into backend storage.
Wrap command in MULTI and EXEC to delete the key from Redis

Pseudocode：

redis.send('WATCH', key)
data = redis.get(key)
database.save(key, data)
redis.send('MULTI')
redis.send('DEL', key)
redis.send('EXEC')

Resource Bundle in IntelliJ TornadoFX Gradle Project

me@iany.me (Ian Yang) — Sat, 03 Jun 2017 16:09:41 +0800

Add properties files into src/main/resources
Global messages location:
- src/main/resources/Messages.properties is for default locale
- src/main/resources/Messages_en_US.properties is for locale en_US
Resource Bundle for component: Same directory structure to the class path. For example the resource bundle file for views.Main can be found in src/main/resources/views/Main.properties, or file name Main_en_US.properties for specific locale.
Enable native-to-ascii conversion in IntelliJ to ease editing UTF-8 values.

Capture Anything using macOS Javascript automation

me@iany.me (Ian Yang) — Sun, 19 Mar 2017 17:54:08 +0800

I have written many scripts to automate the work in macOS. This one is the most frequently used one. The script can capture the current selection in frontend most app in OmniFocus, and I can jump back to the app using URL.

It only supports apps I frequently use. New app support can be added in capturers map.

var tasks = [];
var app = null;
// Uncomment following line to debug a specific app.
// var app = Application("Evernote");
(function(app, tasks) {
var bundleIdentifier;
if (app === null || app === undefined) {
var systemEvents = Application("System Events");
var bundleIdentifier = systemEvents.processes.whose({ frontmost: true })[0].bundleIdentifier();
app = Application(bundleIdentifier);
} else {
bundleIdentifier = app.id();
}
if (tasks === null || tasks === undefined) {
tasks = [];
}
var capturers = {
Safari: function(app, tasks) {
var tab = app.windows[0].currentTab();
tasks.push({ name: tab.name(), note: tab.url() });
},
Chromium: function(app, tasks) {
var tab = app.windows[0].activeTab();
tasks.push({ name: tab.title(), note: tab.url() });
},
Finder: function(app, tasks) {
app.selection().forEach(function(file) {
var name = file.name();
var url = file.url();
var directory = file.container().url();
tasks.push({ name: "[File] " + name, note: url + " in directory " + directory });
});
},
Contacts: function(app, tasks) {
app.selection().forEach(function(contact) {
tasks.push({ name: "Contact " + contact.name(), note: "addressbook://" + contact.id() });
});
},
Evernote: function(app, tasks) {
app.selection().forEach(function(note) {
var url = note.noteLink();
var title = note.title();
tasks.push({ name: "[Evernote] " + title, note: url });
});
},
Mail: function(app, tasks) {
app.selection().forEach(function(mail) {
var url = encodeURI("message://<" + mail.messageId() + ">");
var title = mail.sender() + ": " + mail.subject();
tasks.push({ name: title, note: url });
});
}
};
capturers["Google Chrome"] = capturers.Chromium;
var fluidCapturer = function(app, tasks) {
var tab = app.browserWindows[0].selectedTab();
var name = "[" + app.name() + "] " + tab.title();
tasks.push({ name: name, note: tab.url() });
};
var defaultCapturer = function(app, tasks) {
app.includeStandardAdditions = true;
var url = encodeURI("file://" + app.pathTo().toString());
tasks.push({ name: app.name(), note: url })
};
var capture = capturers[app.name()];
if (capture === undefined) {
if (bundleIdentifier.startsWith("com.fluidapp.FluidApp.")) {
capture = fluidCapturer;
} else {
capture = defaultCapturer;
}
}
capture(app, tasks);
return tasks;
})(app, tasks);
// Then send tasks to OmniFocus or other app which supports Applescript automation.
var of = Application("OmniFocus");
var inbox = of.quickEntry().inboxTasks;
tasks.forEach(function(task) {
inbox.push(of.Task(task));
});
of.quickEntry().open();

I use Keyboard Maestro to invoke the script using keyboard shortcut.

Vcpkg static linking

me@iany.me (Ian Yang) — Sun, 05 Mar 2017 12:13:07 +0800

Vcpkg is a tool published by Microsoft, which is used to manage C/C++ libraries in Windows. It makes libraries installation easier, and it works well with CMake.

But it is not straitforward to staticly link the depdendent libraries using vcpkg.

Under Linux，static linking just need to use .a files instead of .so files (similar in macOS). Use boost as an example, the bundled FindBoost returns found static library files in Boost_LIBRARIES when setting following flags.

set(Boost_USE_STATIC_LIBS ON)
set(Boost_USE_STATIC_RUNTIME ON)

Then just use Boost_LIBRARIES in link_libraries or target_link_libraries to statically link boost.

But when these two flags are set in Windows using vcpkg, CMake will complain that it cannot find static libraries of boost. It seems a constraint of VC compiler, so vcpkg requires install the static version of libraries explicitly. For example, to install boost static 64bit libraries:

vcpkg install zlib:x64-windows-static

And VCPKG_TARGET_TRIPLET is required to generate VC project:

cmake .. \
-DCMAKE_TOOLCHAIN_FILE=/path/to/vcpkg.cmake \
-DVCPKG_TARGET_TRIPLET=x64-windows-static \
-G "Visual Studio 14 2015 Win64"

But the project generated throws error in linking. It can be fixed referring this Stack Overflow thread: CMake compile with /MT instead of /MD. Here is the final CMake sample:

cmake_minimum_required(VERSION 3.0)
project(static_link_test CXX)
set(CMAKE_CXX_STANDARD 11)
if(MSVC)
string(REPLACE "/MD" "/MT" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
string(REPLACE "/MD" "/MT" CMAKE_CXX_FLAGS_DEBUG ${CMAKE_CXX_FLAGS_DEBUG})
string(REPLACE "/MD" "/MT" CMAKE_CXX_FLAGS_RELEASE ${CMAKE_CXX_FLAGS_RELEASE})
else(MSVC)
set(Boost_USE_STATIC_LIBS ON)
set(Boost_USE_STATIC_RUNTIME ON)
endif(MSVC)
find_package(Boost 1.36.0 REQUIRED COMPONENTS filesystem program_options)
include_directories(
${CMAKE_CURRENT_SOURCE_DIR}/src
${Boost_INCLUDE_DIRS}
)
link_libraries(
${Boost_LIBRARIES}
)
add_executable(static_link_test
src/main.cpp
)
install(TARGETS static_link_test RUNTIME DESTINATION bin)

Deploy Concourse CI using Docker

me@iany.me (Ian Yang) — Sat, 25 Feb 2017 18:46:03 +0800

Concourse CI is an awesome open source continuous integration tool. If you are not using Gitlab, and want to setup a CI server, it is a good choice.

Concourse CI provides Docker image and docker compose sample config. But when I run the hello world example, I have met several problems.

First, my Docker host kernel version is too low, which caused the error:

unknown capability "CAP_AUDIT_READ"

The solution is upgrading kernel. Concourse requires version 3.19+¹. Ubuntu 14.04 can use following command:

sudo apt-get install --install-recommends linux-generic-lts-wily

Second, if Concourse is deployed via docker compose, DNS is changed to loopback address. The job will fail because of DNS lookup error:

lookup registry-1.docker.io on 127.0.0.11:53: read udp 127.0.0.1:54286->127.0.0.11:53: read: connection refused

Just add DNS config based on official compose sample to solve this error. I have configured custom DNS for concourse-web and concourse-worker. It is also required to setup a DNS for concourse-worker using environment variable CONCOURSE_GARDEN_DNS_SERVER²。

If your cloud server provider has its own DNS servers, log into any host and check /etc/resolv.conf. Use the DNS servers to replace 8.8.8.8 和 8.8.4.4 in the example below.

concourse-db:
image: postgres:9.5
environment:
POSTGRES_DB: concourse
POSTGRES_USER: concourse
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
concourse-web:
image: concourse/concourse
links: [concourse-db]
command: web
ports: ["8080:8080"]
volumes: ["keys:/concourse-keys"]
environment:
CONCOURSE_BASIC_AUTH_USERNAME: admin
CONCOURSE_BASIC_AUTH_PASSWORD: "${CONCOURSE_BASIC_AUTH_PASSWORD}"
CONCOURSE_EXTERNAL_URL: "${CONCOURSE_EXTERNAL_URL}"
CONCOURSE_POSTGRES_DATA_SOURCE: "postgres://concourse:${POSTGRES_PASSWORD}@concourse-db:5432/concourse?sslmode=disable"
dns:
- 8.8.8.8
- 8.8.4.4
concourse-worker:
image: concourse/concourse
privileged: true
links: [concourse-web]
command: worker
volumes: ["keys:/concourse-keys"]
environment:
CONCOURSE_TSA_HOST: concourse-web
CONCOURSE_GARDEN_DNS_SERVER: 8.8.8.8
dns:
- 8.8.8.8
- 8.8.4.4

The last, Concourse does not pull docker images via the docker daemon in docker host machine. So the config registry-mirror is ignored. The config must be added in the resource source config³. It is tedius that cannot be set as a global config.

jobs:
- name: hello-world
plan:
- task: say-hello
config:
platform: linux
image_resource:
type: docker-image
source:
registry_mirror: https://changeme.mirror.aliyuncs.com
repository: alpine
run:
path: echo
args: ["Hello, world!"]

Convert Web Page to Markdown using Ulysses or Bear

me@iany.me (Ian Yang) — Sun, 27 Nov 2016 00:00:00 +0000

Updates

2017-05-09: The bear current version made clipper easier to use.

Ulysses way:

Select content in browser and copy.
Open Ulysses and use menu Edit > Paste from > Rich text to paste
Select pasted text and use menu Edit > Copy as > Markdown to copy

Bear way:

Copy page URL.
Use /grab-url x-callback-url
After the article is grabbed, use menu Edit > Markdown to copy

Ctags for Ansible

me@iany.me (Ian Yang) — Sun, 08 Feb 2015 00:00:00 +0000

Ansible uses YAML to define tasks, playbooks and handlers. If the files follow some conventions, it is easy to index them using Exuberant Ctags.

For example, following ctags options will index all the lines starting with - name: in .yml and .yaml files. Save it as .ctags in the playbook or role top directory.

--langdef=ansible
--langmap=ansible:.yml.yaml
--regex-ansible=/^[ \t]*-[ \t]*name:[ \t]*(.+)/\1/k,tasks/
--languages=ansible,ruby,python

It is assumed that playbook and task files use extension .yml or .yaml. And name field just follows -.

- name: name must be the first field just following the dash
hosts: all
roles: [ site ]

Now generate the tags

ctags -R .

And try it in vim

:tselect /keyword

To make it works with ctrlp CtrlPBufTag, add following config in vimrc

let g:ctrlp_buftag_types = {
\ 'yaml' : '--languages=ansible --ansible-types=k',
\ }

Keystroke Sequence Shortcuts in Mac OS X

me@iany.me (Ian Yang) — Mon, 13 Oct 2014 00:00:00 +0000

It is a headache to find an available keyboard shortcuts in Mac OS X. I used Option + Letter and Shift + Option + Letter before, since they are preserved for inputting special characters. It has some problems:

Emacs and terminal require a modifier for Meta. I chose Command. It means if I want to use application shortcut with Command in these applications, such as Command+Q to quit the application, I have to use the right one.
I always forget the shortcuts. Although I have listed them in a sheet, it is a pain to keep it synchronized with the shortcuts defined every where.

I switched to a new solution using keystroke sequence shortcuts recently. All my global shortcuts start with Command+M (⌘M). A menu is displayed when I typed prefix. If I forget the shortcut, I just need to glance through the menu.

The shortcuts menu shows all available commands and the corresponding keyboard binding.

Disable System Default Shortcut

⌘M is a system default shortcut to minimize window. I never use it, and it is annoyed when typed by acident. So I disable it and use it as my personal keytroke sequence shortcuts prefix.

Run the following code¹ in terminal to disable it.

defaults write -g NSUserKeyEquivalents -dict-add 'Minimize' '\0'

The changes are applied only in new opened apps. Please reopen all applications, or just restart the system.

Keyboard Maestro

The shortcuts and the menu in the snapshot are managed by Keyboard Maestro. There are two ways to implement keystroke sequence in Keyboard Maestro.

The first solution is setting the same keyboard shortcut for multiple macros. Keyboard Maestro auto detects conflicts and display a menu like the one above. It strips the common prefix, and use the first distinct letter in macro name as the shortcut. Thus, it does not support modifier (like Shift, Command) in subsequent shortcut. And it requires some trick to name your commands, such as “l) Read Later” and “v) Clipboard History”.

Since I want to use modifier, I use another solution: a group with the option “Shows a palette for one action when the hot key xxx is pressed”.

Shows a pallete for one action when the hot key xxx is pressed.

It is tricky to display the shortcuts and align them to the right in the menu like in the snapshot. The shortcuts should be added in command name manually and are aligned by inserting tabs and spaces.

Shortcuts Forwarding

Some application provides global shortcuts, but they cannot be used in Keyboard Maestro. The workaround is using a rarely used and hard to type in the application preference, and forward using a simple shortcut through Keyboard Maestro.

For example, I defined the Alfred 2 clipboard manager shortcut to Ctrl+Option+Shift+Command+F4. I use Command+M, Command+V (Hold Command, type M, then V, and release Command) to show it, because I have a macro in the Command+M group which uses the action “Type a Keystroke”.

Send a global shortcut in an application from Keyboard Maestro by sending a keystroke.

http://apple.stackexchange.com/a/73957

Auto Toggle MacBook Internal Keyboard

me@iany.me (Ian Yang) — Fri, 17 Jan 2014 00:00:00 +0000

I prefer using external keyboard with my MacBook. When no external monitors are used, a typical setup is placing the keyboard above the internal one, so I can still use the internal touchpad.

But sometimes the external keyboard may press some keys of the internal keyboard. There is a solution to disable the internal keyboard, but it is tedious to run the command manually.

# Disable, ignore the warning
sudo kextunload /System/Library/Extensions/AppleUSBTopCase.kext/Contents/PlugIns/AppleUSBTCKeyboard.kext/
# Enable
sudo kextload /System/Library/Extensions/AppleUSBTopCase.kext/Contents/PlugIns/AppleUSBTCKeyboard.kext/

Fortunately, Keyboard Maestro supports executing scripts when a USB device is attached or detached.

I have created two macros to enable and disable internal keyboard.

Before the macros can be used, you must setup sudoers to allow running the command without password.

$ sudo visudo

And append following line to the file and save it.

%admin ALL = NOPASSWD: /sbin/kextunload /System/Library/Extensions/AppleUSBTopCase.kext/Contents/PlugIns/AppleUSBTCKeyboard.kext, /sbin/kextload /System/Library/Extensions/AppleUSBTopCase.kext/Contents/PlugIns/AppleUSBTCKeyboard.kext

Then restart sudo session

$ sudo -K

ActiveRecord includes and preload

me@iany.me (Ian Yang) — Fri, 23 Aug 2013 00:00:00 +0000

ActiveRecord has two query methods to eager load associations, includes and preload. Although the documentation of preload says

Allows preloading of args, in the same way that includes does.

Indeed the two methods have some differences.

In simple words, prefer includes to eager load associations. Use preload only when

you want to customize select columns, or
you meet error “Can not eagerly load the polymorphic association”.

Both of the two methods eager load associations to avoid N+1 queries. However, includes is more aggressive. If it detects that the associations tables are already joined, through explicitly joins or implicit references in where, it overwrites select fields from the joined query and construct result set from it, rather than loading associations in a separate query. See the generated SQL statement below and the fields aliases like t0_r0.

irb(main):049:0> User.includes(:articles).joins(:articles)
SQL (0.7ms) SELECT "users"."id" AS t0_r0,
"users"."login" AS t0_r1,
"users"."name" AS t0_r2,
"users"."created_at" AS t0_r3,
"users"."updated_at" AS t0_r4,
"articles"."id" AS t1_r0,
"articles"."title" AS t1_r1,
"articles"."content" AS t1_r2,
"articles"."user_id" AS t1_r3,
"articles"."created_at" AS t1_r4,
"articles"."updated_at" AS t1_r5
FROM "users"
INNER JOIN "articles"
ON "articles"."user_id" = "users"."id"

However, if the table is not joined yet, includes will fallback to preload, which loads association in a separate query. See the SQL in log below that articles are eager loaded in another SELECT query.

irb(main):050:0> User.includes(:articles)
User Load (0.1ms) SELECT "users".* FROM "users"
Article Load (0.2ms) SELECT "articles".* FROM "articles"
WHERE "articles"."user_id" IN (1, 2)

Because includes may (or may not) overwrite select, if you have your own select clause, use preload instead. See example below:

users = User.joins(:articles).select(
'users.*, articles.created_at as last_posted_at'
)
users.includes(:articles).first.last_posted_at
# NoMethodError: undefined method `last_posted_at'
users.preload(:articles).first.last_posted_at
# => "2013-08-23 11:20:36.536968"

Older version of Rails has trouble to includes polymorphic associations. It seems the newer version of Rails is smart enough to fallback to preload. However if you still see the error “Can not eagerly load the polymorphic association”, try switching to preload.

References

ActiveRecord uniq, count and distinct

me@iany.me (Ian Yang) — Tue, 16 Jul 2013 00:00:00 +0000

ActiveRecord has two methods to remove duplicates. Method uniq and option distinct: true in method count. I thought uniq.count and count(distinct: true) were identical. Indeed, uniq.count still counts duplicates, and count(distinct: true) must be used here.

In simple words, use uniq to get unique result set, use count(distinct: true) to count unique result.

For example, user has many activities, and I want to get all users having a specific type of activities:

users = User.joins(:activities).where(
activities: { activity_type: 'purchase'}
)

Because a user may have multiple activities with the same type, the result above may contain duplicate users. Method uniq can be used here to remove the duplicates:

users = users.uniq

But users.uniq.count generates SQL like below:

SELECT DISTINCT COUNT(*) ...

This SQL counts all records with duplicates, and apply DISTINCT on the count, which has only one row. So DISTINCT has no effect here.

On the other hand, users.count(distinct: true) generates SQL below, which removes duplicates first, then count the result.

SELECT COUNT(DISTINCT users.id) ...

Responsive SVG

me@iany.me (Ian Yang) — Sun, 16 Jun 2013 00:00:00 +0000

SVG can use percentage as unit, but it is often more convenient to use px as unit. Framework such as d3 also uses px internally. However, it is still easy to scale an SVG using attributes viewBox and preserveAspectRatio even px is used as unit internally.

viewBox and preserveAspectRatio

The attribute viewBox specifies viewpoint dimension. SVG can be scaled along with the parent container without changing the internal coordinates.

For example, the SVG below defines a viewpoint with width 100 and height 100.

<svg viewBox="0 0 100 100">
<circle cx="50" cy="50" r="50" fill="blue">
</svg>

The figure below shows the same SVG in different container sizes: 50x50, 100x50, 100x100, 100x200, 200x200 from left to right.

Same SVG in Different Size Containers

Attribute preserveAspectRatio controls how SVG is scaled according to parent container. There are many combinations, but the most frequently used are default (when the attribute is not specified, or set to xMidYMid meet explicitly) and none.

default, preserve ratio, align center of SVG to the center of the container, scaled up as much as possible but ensure the entire viewBox is visible within the container.
none, do not preserve ratio, scale SVG to fill the whole container.

See the figure similar to the previous one but setting preserveAspectRatio to none.

preserveAspectRatio="none"

Responsive SVG

First add following CSS:

svg {
width: 100%;
height: 100%;
}
svg * {
vector-effect: non-scaling-stroke;
}

The first rule ensures SVG always fills parent container. And the second one sets a default vector-effect that when SVG is scaled, leave stroke width untouched.

If the SVG does not need to preserve ratio, set preserveAspectRatio to none, and set the height of the container. For example, the SVG below has a fixed height 50px, and is half width of the whole page.

<div style="width:50%;height:50px;">
<svg viewBox="0 0 100 100" preserveAspectRatio="none">
<circle cx="50" cy="50" r="50" fill="blue">
</svg>
</div>

Fixed With, Horizontal scalable

If the SVG must preserve ratio, then we need a bit of JavaScript to change container height when window is resized.

$.fn.preserveAspectRatio = function() {
this.each(function() {
var $this = $(this);
var ratioSpec = $this.attr('data-preserveAspectRatio').split(':');
var ratio = parseInt(ratioSpec[1], 10) / parseInt(ratioSpec[0], 10);
$this.height($this.width() * ratio);
});
};
var preserveAspectRatio = function() {
$('[data-preserveAspectRatio]').preserveAspectRatio();
};
$(function() {
preserveAspectRatio();
$(window).on('resize', preserveAspectRatio);
// Better to wrap preserveAspectRatio in underscore debounce or jquery-throttle-debounce.
// $(window).on('resize', _.debounce(preserveAspectRatio, 200));
});

Then set data-preserveAspectRatio on any container that needs preserve aspect ratio. Following example sets the width height ratio to 1:1:

<div style="width:50%;" data-preserveAspectRatio="1:1">
<svg viewBox="0 0 100 100" preserveAspectRatio="none">
<circle cx="50" cy="50" r="50" fill="blue">
</svg>
</div>

See the demo in this pen.

References

How Rails Assets Prefix Disables the Session

me@iany.me (Ian Yang) — Sun, 28 Apr 2013 00:00:00 +0000

This is original posted on intridea blog.

I recently worked in a Rails project with Peter (@sporkd). The project is intended to be used as a sub-site, and should be served under sub-URI. After google, we ended up by setting config.assets.prefix and wrapped all routes in scope. The solution is simple and worked well. But soon, some weird bugs were found, and Peter was successfully isolated the problem to session (see demo sporkd/asset_prefix_test)

After several hours debugging, I finally found out the cause. To make a long story short, the routes configured in routes.rb should not start with config.assets.prefix, otherwise session save is skipped. The demo sporkd/asset_prefix_test can be fixed by simply setting config.assets.prefix to /one/assets. You also get a bonus by setting a unique prefix for assets, since it is easy to add expiration header for assets in web server.

X-Cascade Header in Rails

I never knew X-Cascade header in Rails before. @soulcutter has a post that described its usage.

The basic idea is this: if you return a response from a controller with the X-Cascade header set to “pass”, it indicates that your controller thinks something else should handle the request. So rails (or is it rack? in any case…) will continue down your routes looking for the next rule that matches the request.

Indeed, X-Cascade is not only restricted in controller, if a mounted engine sets this header, Rails also continues down the routes searching.

It is a feature of Rails. Since 3.2, Rails has moved the routes logic to journey. The X-Cascade trick can be found in journey/router.rb#L69.

Pay attention that, the rack env object is shared when request is passed on. So if env is changed by former route, the latter one is affected. This is the root cause of the weird session issue, because session is controlled by env['rack.session.options'].

Sprockets, who skips the session

Sprockets, the gem for rails assets pipeline, mounts itself on config.assets.prefix and prepends the route to Rails. So if user accesses a page which path starts with config.assets.prefix, sprockets always processes the request first.

Maybe for performance, sprockets disables session save by changing env['rack.session.options']:

env['rack.session.options'] ||= {}
env['rack.session.options'][:defer] = true
env['rack.session.options'][:skip] = true

The options are changed even when asset is not found. If so, sprockets returns 404 and sets the header X-Cascade. Then Rails passes the request to controller, and correct page is rendered as expected. However, since the session is already disabled by sprockets, the changed session in controller is never saved.

Because env is a shared resource between routes when X-Cascade is set, it should not be changed unless it has to. When asset is not found, sprockets should just pass though without touching env, so I submit a PR for it.

How we Debug

Peter and I worked in different time zones. He first found the session issue because several features related to session did not work. He made the demo sporkd/asset_prefix_test to isolate the issue using minimum code at the end of the day in his time zone and left me the message.

When my day started, I got the message and started debugging on session based on the demo in doitian/asset_prefix_test.

Because session store class is customizable, I inherited one from default cookie store and added breakpoints using pry. Soon I found out that options[:skip] was true, but I had no idea where it was set to true. Then I did a grep (using ag) in all gems, and fortunately, only sprockets has set this option to true. The remaining work was just figuring out why sprockets is invoked before controller action.

Select multiple zsh completions item

me@iany.me (Ian Yang) — Mon, 08 Apr 2013 00:00:00 +0000

# ^Xa to insert all matches
zle -C all-matches complete-word _generic
bindkey '^Xa' all-matches
zstyle ':completion:all-matches:*' old-matches only
zstyle ':completion:all-matches::::' completer _all_matches

UPDATE: The snippet above brings insert-completions to zsh. Thanks, Trevor Joynson, who has provided the solution in the comment.

Command insert-completions (see man bash) is the feature I most miss in bash.

[insert-completions] Insert all completions of the text before point that would have been generated by possible-completions.

Its default binding is M-*, a.k.a, Alt-Shift-8 or Meta-Shift-8.

It is useful for simple batch operations, such as removing all git branches starting with feature/.

After I switched to zsh, I have been looking for a similar command, but for no results. However, I just discovered the zsh way to select multiple completions recently, using accept-and-hold (see man zshzle).

[accept-and-hold] Push the contents of the buffer on the buffer stack and execute it.

The default binding is M-a (a.k.a, Alt-a or Meta-a). If the command line has input content ls, M-a executes the line ls without clearing the input.

It seems unrelated. But if completion menu is activated and a menu item is selected (the simplest way is Tab twice), accept-and-hold accepts the current menu item without closing completing menu. So it is possible to select multiple items by navigating to wanted items, and type M-a to select them.

Bash Sample to delete git branches

First enter following text.
```
$ git branch -D feature/
```

Type Tab to activate completion.

$ git branch -D feature/
feature/a feature/b

Type M-* to insert all completions.
```
$ git branch -D feature/a feature/b
```
Review, delete branches should be keep.
Type Enter to execute the line.

Zsh way

First enter following text.
```
$ git branch -D feature/
```

Type Tab to activate completion.

$ git branch -D feature/
feature/a feature/b

Type Tab again, to select first item. Brackets demonstrate which item is currently selected.
```
$ git branch -D feature/a
[feature/a] feature/b
```

Type M-a to keep selected item and continue completing

$ git branch -D feature/a feature/b
feature/a [feature/b]

Use Tab to skip branches that should be keep. The item selection mode also supports C-n, C-p, C-f, C-b to move around, and C-s, C-r to search.
When cursor is on the last branch to be deleted, type Enter to accept it and stop completion. If M-a is typed by accident, you have to delete the last inserted completion item, because zsh auto inserts current selected item.

Discover zsh commands

The builtin bindkey lists all commands with key-bindings. They are most frequently used commands, that’s why they deserve a key-binding.

The commands reference can be looked up in man zshzle. I also discovered several other interesting key-bindings:

M-h Run man on current entered command. Say you are editing tcpdump options, and want to check manual without losing the input.
M-? Run which-command on current entered command.

A trick to use just jasmine gem to test Javascript in Rails

me@iany.me (Ian Yang) — Thu, 31 Jan 2013 00:00:00 +0000

Jasmine is a behavior-driven development framework for testing JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM. And it has a clean, obvious syntax so that you can easily write tests.

It is very easy to integrate Jasmine into Rails, since the team provides the jasmine gem. The jasmine gem also supports assets pipeline, just prepend assets/ to file path. For example, app/assets/javascripts/application.js.coffee can be referred as assets/application.js in jasmine.yml src_files.

However, spec_files does not support assets pipeline, so the files in spec/javascripts cannot be written in CoffeeScript. But if spec/javascripts is appended to assets pipeline paths, the spec files can be added to src_files in jasmine.yml.

This post explains how to apply such trick, and how to integrate jasmine gem with guard-jasmine and travis.

I also created a demo repository, see its commits for integration steps.

Jasmine Integration

Jasmine 1.3.1 removes the ability to load jasmine_config.rb before executing specs. Although it is possible to load the file by adding a rake dependency, but the trick does not work for rake jasmine:ci. So please use Jasmine 1.3.0 until a new version is released.

First add jasmine in Gemfile

group :development, :test do
gem "jasmine", "1.3.0"
end

and run bundle install.

Then generate config files:
```
rails g jasmine:install
```
Append spec/javascripts to assets paths by creating file spec/javascripts/support/jasmine_config.rb and add following line to the file:
```
Rails.application.assets.append_path File.expand_path('../..', __FILE__)
```
Edit spec/javascripts/support/jasmine.yml. Append assets/specs.js to src_files. Also set spec_files and helpers to [], otherwise the JavaScript files may be included twice.
```
src_files:
- assets/application.js
- assets/specs.js
stylesheets:
- stylesheets/**/*.css
helpers: []
spec_files: []
src_dir:
spec_dir:
```

Create the file spec/javascripts/specs.js. Usually, it just needs include all the js files in directory spec/javascripts.

// Ensure helpers are loaded first. Remove the following line if
// helpers directory is not created yet.
//= require_tree ./helpers
//= require_tree ./

Create a spec file to test the integration, e.g., spec/javascripts/foobar_spec.js.coffee

# spec/javascripts/foobar_spec.js.coffee
describe "foobar", ->
it 'works', -> expect(1 + 1).toEqual(2);

Start jasmine server by rake jasmine and visit the test page http://localhost:8888.

Guard

The defult config of guard-jasmine is intended to be used with jasminerice. But it also allows starting jasmine gem server as well.

Add guard and guard-jasmine in Gemfile.

group :development do
gem 'guard'
gem 'guard-jasmine'
end

bundle install

Add following jasmine guard config in Guardfile

require 'guard/jasmine'
port = ::Guard::Jasmine.find_free_server_port
guard :jasmine, :server => :jasmine_gem, :port => port, :jasmine_url => "http://localhost:#{port}/" do
watch(%r{spec/javascripts/support/.+\.(?:rb|yml)$}) { 'spec/javascripts' }
watch(%r{spec/javascripts/helpers/.+\.rb$}) { 'spec/javascripts' }
watch(%r{spec/javascripts/spec\.(?:js\.coffee|js|coffee)$}) { 'spec/javascripts' }
watch(%r{spec/javascripts/.+_spec\.(?:js\.coffee|js|coffee)$})
watch(%r{app/assets/javascripts/(.+?)\.(js\.coffee|js|coffee)(?:\.\w+)*$}) { |m| "spec/javascripts/#{ m[1] }_spec.#{ m[2] }" }
end

Now guard-jasmine will start the server provided by jasmine-gem and visit the test page using phantomjs.

CI

The jasmine gem has another task rake jasmine:ci for continuous integration environments. To run the test on server without GUI, xvfb can be used.

See the following .travis.yml, in before_install, xvfb is started and DISPLAY is set so GUI applications know where to render their window.

language: ruby
script: bundle exec rake jasmine:ci
before_install:
- "/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -ac -screen 0 1280x1024x16"
- "export DISPLAY=:99.0"
- "export JASMINE_BROWSER=firefox"

References

Rails Compound Input

me@iany.me (Ian Yang) — Sat, 19 Jan 2013 00:00:00 +0000

When I implement time input feature for 19wu (an open source ticket sale system), I want to split the datetime into date and time parts, so JavaScript date picker and time picker can be used. This post introduces two methods I found.

Compound datetime input

composed_of utilizes assign_multiparameter_attributes trick like datetime_select, and fields_for mocks an association.

The github repository doitian/rails-compound-input-demo contains demos for both methods.

composed_of

Rails has built-in compound inputs date_select, time_select and datetime_select. They use trick that if parameter name has parentheses, they will be used in the attribute class constructor (see assign_multiparameter_attributes). However, datetime attribute is of type DateTime, which accepts 6 parameters year, month, day, hour, minute and second.

composed_of can be used here to represent the datetime attribute using value object, which class constructor accepts date and time strings. See CompoundDatetime#initialize below.

※ compound_datetime.rb

class CompoundDatetime
def self.from_datetime(datetime)
new.tap do |result|
result.datetime = datetime
end
end
attr_accessor :datetime
# Accepts date and time string. The form just need to submit params
#
# - compound_beginning_time(1s) for date
# - compound_beginning_time(2s) for time
def initialize(date = nil, time = nil)
if date.present?
@datetime = Time.zone.parse([date.presence, time.presence || ''].join(' '))
end
end
def date
@datetime.strftime('%Y-%m-%d') if @datetime
end
def time
@datetime.strftime('%H:%M') if @datetime
end
end

Then setup the mapping in model Event:

※ event.rb

class Event < ActiveRecord::Base
attr_accessible :beginning_time, :title
attr_accessible :compound_beginning_time
composed_of :compound_beginning_time, {
:class_name => 'CompoundDatetime',
:mapping => [ %w(beginning_time datetime) ],
:converter => Proc.new { |datetime| CompoundDatetime.from_datetime(datetime) }
}
end

The form view just needs set correct name:

※ events/_form.html.erb

<div class="field">
<%= f.label :compound_beginning_time, 'Begining Time' %><BR />
<%= text_field_tag 'event[compound_beginning_time(1s)]', @event.compound_beginning_time.date, :placeholder => 'yyyy-mm-dd' %>
<%= text_field_tag 'event[compound_beginning_time(2s)]', @event.compound_beginning_time.time, :placeholder => 'HH:MM' %>
</div>

fields_for

fields_for is usually used to embed associations in form. However, all it required was a method to return the named attribute, and then a <field>_attributes= writer to interpret the hash on the other side. See the section “Nested Attributes Examples” in the fields_for API document.

First create CompoundDatetime class which exposes date and time fields. assign_attributes handles the hash params passed from form. Method persisted? is required to quiet NoMethodError.

※ compound_datetime.rb

class CompoundDatetime
attr_accessor :datetime
def initialize(datetime)
@datetime = datetime
end
# accepts hash like:
#
# {
# 'date' => '2012-12-20',
# 'time' => '20:30'
# }
def assign_attributes(hash)
if hash[:date].present?
@datetime = Time.zone.parse([hash[:date].presence, hash[:time].presence || ''].join(' '))
end
self
end
def date
@datetime.strftime('%Y-%m-%d') if @datetime
end
def time
@datetime.strftime('%H:%M') if @datetime
end
def persisted?; false; end
end

The model just delegates the named attribute and <field>_attributes= method to CompoundDatetime.

※ event.rb

class Event < ActiveRecord::Base
attr_accessible :beginning_time, :title
attr_accessible :compound_beginning_time_attributes
def compound_beginning_time
CompoundDatetime.new(beginning_time)
end
def compound_beginning_time_attributes=(attributes)
self.beginning_time = compound_beginning_time.assign_attributes(attributes).datetime
end
end

The form view uses fields_for helper to nest fields of compound_begining_time.

※ events/_form.html.erb

<div class="field">
<%= f.label :beginning_time %><br />
<%= f.fields_for :compound_beginning_time do |fields| %>
<%= fields.text_field :date, :placeholder => 'yyyy-mm-dd' %>
<%= fields.text_field :time, :placeholder => 'HH:MM' %>
<% end %>
</div>

Reference

Fix TAB Binding For yasnippet And auto-complete

me@iany.me (Ian Yang) — Sat, 31 Mar 2012 00:00:00 +0000

There are two TAB’s in Emacs, (kbd "TAB") (\t, [9]) and (kbd "<tab>") ([tab]). If modes like yasnippet and auto-complete want to bind on TAB, their trigger key must be the same with the original Tab command. Since Emacs binds indent-for-tab-command on (kbd "TAB"), so it’s better to use it as the trigger key. Yasnippet binds to it by default, It is also easy to setup auto-complete to trigger using Tab.

;; trigger using TAB and disable auto start
(custom-set-variables
'(ac-trigger-key "TAB")
'(ac-auto-start nil)
'(ac-use-menu-map t))

But in some modes (ruby-mode, markdown-mode, org-mode), the command is bind to (kbd "<tab>"), when the real Tab key is typed, the function bind on (kbd "<tab>) has higher priority, so yasnippet and auto-complete are not invoked. It is easy to fix by moving the keybinding:

(defun iy-tab-noconflict ()
(let ((command (key-binding [tab]))) ; remember command
(local-unset-key [tab]) ; unset from (kbd "<tab>")
(local-set-key (kbd "TAB") command))) ; bind to (kbd "TAB")
(add-hook 'ruby-mode-hook 'iy-ac-tab-noconflict)
(add-hook 'markdown-mode-hook 'iy-ac-tab-noconflict)
(add-hook 'org-mode-hook 'iy-ac-tab-noconflict)

Use Popup isearch For Yasnippet Prompt

me@iany.me (Ian Yang) — Fri, 30 Mar 2012 00:00:00 +0000

Yasnippet tries functions in yas/prompt-functions when it needs user to select one choice, such as selecting snippets with the same trigger key, such as helper method yas/choose-value.

popup is a visual popup interface library extracted from auto-complete by its author. It has better look and feel than all the built-in yas/prompt-functions. Also it is easy to customize, and its isearch mode is very efficient, the items are filtered on-the-fly when typing.

The integration is easy. Load popup.el, implement one prompt function and add it to yas/prompt-functions.

※ yasnippet-popup-isearch-prompt.el

(require 'popup)
(require 'yasnippet)
;; add some shotcuts in popup menu mode
(define-key popup-menu-keymap (kbd "M-n") 'popup-next)
(define-key popup-menu-keymap (kbd "TAB") 'popup-next)
(define-key popup-menu-keymap (kbd "<tab>") 'popup-next)
(define-key popup-menu-keymap (kbd "<backtab>") 'popup-previous)
(define-key popup-menu-keymap (kbd "M-p") 'popup-previous)
(defun yas/popup-isearch-prompt (prompt choices &optional display-fn)
(when (featurep 'popup)
(popup-menu*
(mapcar
(lambda (choice)
(popup-make-item
(or (and display-fn (funcall display-fn choice))
choice)
:value choice))
choices)
:prompt prompt
;; start isearch mode immediately
:isearch t
)))
(setq yas/prompt-functions '(yas/popup-isearch-prompt yas/no-prompt))

CSS Line Wrap Indicator

me@iany.me (Ian Yang) — Sat, 04 Feb 2012 00:00:00 +0000

Many editor can wrap a line that reaches the window width and show an indicator in the margin, for example, Emacs.

Emacs line wrap with bent arrow in fringe

I want to add such line wrap indicators to the code block in HTML. It is easy to add indicators as background image in CSS. But the indicators should only be shown when the line is wrapped. And from the figure, the wrap indicator is not displayed on the last line in the right margin, and not on the first line in the left margin. There’s a :first-line pseudo element selector, there’s no :last-line. However, it can be achieved by using :before and :after with position trick.

You can check the result in this jsfiddle. Resize the result panel to see the line wrap indicators. Following is a detailed explanation.

Add Line Markup

First, the code block in pre must be split by lines and add markup so that CSS can be applied.

<pre><code>(font-lock-add-keywords
'ruby-mode
'(("\\(\\b\\sw[_a-zA-Z0-9]*:\\)\\(?:\\s-\\|$\\)" (1 font-lock-constant-face))))
</code></pre>

is converted to

<pre><code><span class="line">(font-lock-add-keywords
</span><span class="line"> 'ruby-mode
</span><span class="line"> '(("\\(\\b\\sw[_a-zA-Z0-9]*:\\)\\(?:\\s-\\|$\\)" (1 font-lock-constant-face))))
</span></code></pre>

It is easy to do such conversion by replace new line character \n to \n</span><span class="line"> and then wrap with first open tag and the last close tag.

Because I want to show indicators in left, right padding of span.line, it should be displayed as block to expand horizontally. Also enable line wrap on pre.

pre {
white-space: pre-wrap;
}
pre code, span.line {
display: block;
}

Pay attention to the new line position. It is included in the added span.line elements. Otherwise extra empty line is added between lines.

Add Indicators

The indicators are added through :before and :after. They have the same height with the line by setting height to 100%.

To ensure the indicators are aligned to each wrapped line, the picture must be the same height with line-height.

I use following two icons. The height is 28px. So also set span.line line-height to 28px.

Left:
Right:

span.line {
line-height: 28px;
}

Then just add :before and :after and apply vertically repeat background on them. position of span.line is set to relative so :before and :after can be positioned relative to it. It is easy to add them in span.line padding. I use padding instead of margin because later I’ll set overflow to hidden to hide indicators outside of span.line, where margin is considered outside.

span.line {
padding: 0 13px; /* 8px for indicator, 5px for space around text */
position: relative;
}
span.line:before, span.line:after {
background-repeat: repeat-y;
content: "";
display: block;
width: 10px;
height: 100%;
position: absolute;
}
span.line:before {
background-image: url("line-left.png");
top: 0;
left: 1px;
}
span.line:after {
background-image: url("line-right.png")
top: 0;
right: -1px;
}

Now the indicators are shown in left and right padding on all lines. Use top to shift down left indicators and bottom to shift up right indicators, so left indicator is not shown on the first line and right indicator is not shown on the last line. But some indicators are moved out, use overflow:hidden to hide them.

span.line {
overflow: hidden;
}
span.line:before {
top: 28px;
}
span.line:after {
top: auto;
bottom: 28px;
}

Add Color to Padding

Use border and background on pre to add different color to indicators padding and code block. Use negative margin to align indicators to the borders.

pre {
border-style: solid;
border-color: #EEE;
border-width: 0 8px;
background-color: #AAA;
}
span.line {
margin: 0 -8px;
}

Highlight Ruby New Hash In Emacs

me@iany.me (Ian Yang) — Thu, 12 Jan 2012 00:00:00 +0000

Not fully tested, let me know if it mess up your buffer.

※ my-ruby-mode.el

(font-lock-add-keywords
'ruby-mode
'(("\\(\\b\\sw[_a-zA-Z0-9]*:\\)\\(?:\\s-\\|$\\)" (1 font-lock-constant-face))))

Zero Configuration Nginx With Passenger

me@iany.me (Ian Yang) — Fri, 09 Dec 2011 00:00:00 +0000

Pow is a zero-config Rack server for Mac OS X. Here I steal its idea to build a zero-config system based on Nginx with Passenger.

Auto Virtual Hosts

Since 0.8.25, Nginx supports named captures in server_name directive.

server {
server_name ~^(www\.)?(?<domain>.+)$;
root /sites/$domain;
}

It can be used to setup passenger virtual hosts and locate the rails/rack root directory based on the server name.

server {
server_name ~^(.*\.)?(?<app>[^.]+)\.dev$;
root /opt/apps/$app/public;
rails_env development;
passenger_enabled on;
}

The configuration above matches all sever names that ends with “.dev”, and use the 2nd level domain name to locate application root directory. For example, myapp.dev and www.myapp.dev both use the root /opt/apps/myapp/public, thus will serve the rails/rack app /opt/apps/myapp/.

Sure, /opt/apps can be changed to any directory accessible by Nginx. And remember to reload Nginx (nginx -s reload) after change the config file.

Now new app can be added just like Pow, creating a symbol link in /opt/apps.

cd /opt/apps
ln -s /path/to/myapp

We can setup more server blocks for different environments, for example:

server {
server_name ~^(.*\.)?(?<app>[^.]+)\.staging;
root /opt/apps/$app/public;
rails_env staging;
passenger_enabled on;
}

Add a record in /etc/hosts and open browser to see whether http://myapp.dev works.

127.0.0.1 myapp.dev

Local DNS Resolver

Pow starts its own DNS and utilizes Mac /etc/resolver system to resolve all subdomains of specified top-level domain to 127.0.0.1.

It is written for nodejs, and I wrote a simple wrapper simpledns to only start the DNS server in Pow.

Install nodejs first

Clone simpledns

git clone git://github.com/doitian/simpledns.git

Read README.md in it and start the DNS server.

The server echoes something like

Save following content as files: /etc/resolver/dev
# File generated by simpledns/server.js
namespace: 127.0.0.1
port: 20560
# File content ends here

Follow the instruction and create the file in /etc/resolver.

Now remove the line 127.0.0.1 myapp.dev, restart your browser to see whether http://myapp.dev still works.

[Outdated] Tmux And Rvmrc

me@iany.me (Ian Yang) — Mon, 27 Jun 2011 00:00:00 +0000

RVM has since moved to using .profile, so just put the “cd .” in .profile and it will work

— Mikael Wikman commented below

Original Article

Tmux is a terminal multiplexer. I switched to Tmux from GNU Screen recently.

I work on several Ruby projects. I use RVM to manage gem set for different projects, and use rvmrc file to switch gem set automatically. I usually start a Tmux session for a project in its root directory, so all the windows and panes in the session use the project root as default directory. The problem is, the new created shell in the session does not load .rvmrc in the root directory. I have to force loading the file using “cd .”

I also use Tmuxinator. It can specify a rvm gem set though project yml file. But this setting only has effect on the windows and panes pre-configured in the yml file, the new created shells do not load .rvmrc. Moreover, if the project needs different gem sets for different branches, each gem set must has its own tmuxinator project file.

Finally, I figured out a simple solution. Just append cd . to the end of the shell init file (.bashrc for bash and .zshrc for zsh). Then all the new shells will try to load the .rvmrc file in the default directory where it starts, including shells in Tmux.

# try loading .rvmrc, add it below the line loading RVM
cd .

Mendeley: Cross Platform Research Management Tool

me@iany.me (Ian Yang) — Wed, 18 May 2011 00:00:00 +0000

Mendeley is a research management tool for desktop & web. It has clients for Linux, Mac, Windows and iPhone and a Web interface. Mendeley can manage any document, but is better to work with PDF. The file meta data are synchronized though Mendeley server. The attached files (PDF or any other formats) can also be synchronized, but the free account has a quota of 500MB. However, the attached files directory and underlying sqlite database can be synchronized manually.

Mendeley is also a collaboration platform that all the added documents can be shared with others. Co-workers can create group to maintain a shared documents list. Mendeley also maintains a public database, from which you can search and add result to your Mendeley library directly. Mendeley also supports importing data from other services such as CiteULike.

Mendeley client supports tags and PDF file indexing. It makes easy to search documents. If you use BibTex to manage references in your writings, the client can export all documents into a BibTex file and synchronize your further changes in Mendeley to that file.

Get Started

You need to register an account in Mendeley. Then download a client for your platform. Following is the snapshot of Mendeley under Linux.

Mendeley Desktop

Add Documents

The documents can be added by adding attached files directly. Mendeley will try to extract meta information from files. It is also able to create an entry and edit meta information manually, and attach file later.

But the most convenient solution is search the documents in Mendeley database, add the search result and synchronize to the client, then attach files in client. However, I have not found a way to use a search result as the meta information of an existing document.

The meta information and attached files can be managed in details panel, which is opened by clicking the handler in the right side.

Synchronize

Documents meta information are all synchronized. The attached files are synchronized according to the per folder settings. All folders are listed under My Library. Just select the folder you want to synchronize and edit its settings to enable.

If attached files are not synchronize attached files, their local paths are also not synchronized. It is very inconvenient. You cannot synchronize your attached files manually across machines with the same position and let Mendeley just synchronize the meta information.

However, you can manually synchronize the attached files directory and underlying sqlite database.

Organize files

Open options dialog through menu Tools -> Options and switch to “File Organizer” tab.

Enable “Organize my files” and select a directory such as “/Mendeley”. It must be the same in different machines. Otherwise the manual solution cannot synchronize between Windows and Mac, Linux. Since in Windows, the path name contains the driver name, so the database also cannot be shared between Windows and other systems having root path.
Enable “Sort files into sub-folders” to avoid file name conflicts. I prefer the Folder path Year/Author.

File Organizer

Manual Synchronization

The attached files directory and the meta info sqlite database can be synchronized using Dropbox, Rsync or Lsyncd. For Dropbox, just make symbol links in the Dropbox directory.

The database file is created when you have logged-in in the client. It is stored in different location in different platform. I have figured out its location in Linux and Mac (replace email@example.com with your own registered email).

Linux: $HOME/.local/share/data/Mendeley Ltd./Mendeley Desktop/email@example.com@www.mendeley.com.sqlite
Mac: $HOME/Library/Application Support/Mendeley Desktop/email@example.com@www.mendeley.com.sqlite

Because the sqlite database file is binary, you cannot simply merge two database files. I recommend to add and edit documents only in one machine, and use other machines as read only copies. Otherwise you may loose data during synchronization.

Switch Window Using Fuzz Matching

me@iany.me (Ian Yang) — Wed, 25 Aug 2010 00:00:00 +0000

This article demonstrates how to quickly switch to a window using gpicker and wmctrl. You type significant letters of workspace name, application name or title and gpicker provides a list of windows you most likely mean to pick.

Gpicker

Gpicker is a program that allows you to quickly and conveniently pick file in a (possibly very large) project. You type significant letters of file name (typically from the start of words) and gpicker provides you with a list of files you most likely mean to pick. The program filters and orders project’s list of files in real-time as you type.

You need to download and install it manually (for Ubuntu, you can download the Debian package).

Gpicker is designed as a filter, so it can be used to select item from an arbitrary list. The selected item is printed on standard output.

For example, use gpicker to select a number between 1 and 10:

$ seq -w 1 10 | gpicker -n "\n" -

Gpicker picks 1 to 10

The last dash tells gpicker to read list from standard input, and option “-n” sets the list separator. Gpicker uses “\0” as item separator by default.

Wmctrl

Wmctrl is a command line utility to interact with a EWMH/NetWM compatible X Window Manager. It can list all windows, switch to a window or bring a window to current workspace. Wmctrl can be installed by package manager in most Linux distribution.

Wmctrl lists all windows using option “-l”. We can add “-x” to list windows resource and class as well. The output of wmctrl -l -x looks like:

0x03200009 0 urxvt.URxvt ian-desktop urxvt
0x0340008e 1 Navigator.Firefox ian-desktop Vimperator / test
0x03e000a4 2 emacs.Emacs ian-desktop Emacs: _theme.sass

The first column is the window ID. We can switch to or bring a window using the ID.

# switch to Emacs, raise and focus it
wmctrl -i -a 0x03e000a4
# bring Emacs to current workspace, raise and focus it
wmctrl -i -R 0x03e000a4

Windows Picker

Now we can use wmctrl to list windows, gpicker to select one window, and wmctrl again to switch to the selected window.

wmctrl -i -a `wmctrl -l -x | gpicker -n "\n" -d "\n" -`

The directory separator is set to “\n” in the first windows picker above, because gpicker is designed to pick files primarily and it has special rules in the matching algorithm.

First the item is split by directory separator. Before we enter a directory separator ("/" by default), only the last component (it is the file name if the item is a file path) is used in matching. For example, “bin” does not match item “bin/ruby”, but “bin/”, even “b/” does match. If the window title contains the directory separator, for example the Firefox in the wmctrl -l -x output above:

0x0340008e 1 Navigator.Firefox ian-desktop Vimperator / test

To match it, we must enter “firefox/”. So I set directory separator to “\n” to disable this feature, and the whole item is used to match.

However, we can take advantage of this feature. If we construct the window item likes:

id/workspace/window_class/window_title

I can match window title by type significant letters directory. I also can use directory separator “/” to filter all windows in a specific workspace, or windows with the same window class.

It is very useful for me. Because I uses tiling windows manager Xmonad and places windows in different windows. Following are some examples:

I have entered	Windows I want to match
www/ff/	Firefox in workspace “2.www”
ruby	Windows which title contains ruby
xpdf/ruby	A document opened using xpdf and the title contains ruby
sys/usrshare	The terminal I opened visit /usr/share in workspace “1.sys”

Sure, we have to remove “/” from workspace name and window title.

I use a Ruby script to format the list. Workspace name can be listed using wmctrl -d.

Now we can use the ruby script in the revised version:

wmctrl -i -a `wlist.rb | gpicker - | sed 's;/.*$;;'`

Replace “-a” with “-R” to bring window to current workspace.

Study on Alias Method

me@iany.me (Ian Yang) — Sat, 29 May 2010 00:00:00 +0000

@miloyip has published a post recently which motioned the Alias Method to generate a discrete random variable in O(1). After some research, I find out that it is a neat and clever algorithm. Following are some notes of my study on it.

What is Alias Method

Alias method is an efficient algorithm to generate a discrete random variable with specified probability mass function using a uniformly distributed random variable.

Let $Z$ be the discrete random variable which has n possible outcomes $z_0,z_1,\ldots,z_{n-1}$ . To make the discussion below simple, we study another variable $Y$ , where $P\{Y=i\}=P\{Z=z_i\}$ . And when $Y$ takes on value $i$ , let $Z$ be $z_i$ . So $Z$ can be generated from $Y$ .

Random variable $X$ is uniformly distributed in $(0, n)$ , which probability density function is

 $f(x) = \left\{ \begin{array}{rl} 1/n & \text{if } 0 < x < n\\ 0 & \text{otherwise}\\ \end{array} \right.$

Now generate a variable $Y'$ that

 $Y' = \left\{ \begin{array}{rl} \lfloor x \rfloor & \text{if } (x - \lfloor x \rfloor) < F(\lfloor x \rfloor)\\ A(\lfloor x \rfloor) & \text{otherwise}\\ \end{array} \right.$

$A(i)$ is the alias function. When $x$ falls in range $[i, i + 1)$ ( $i$ is an integer), $y$ has the probability $F(i)$ to be $i$ , and probability $1 - F(i)$ to be $A(i)$ . Because $x$ is uniformly distributed,

 $\begin{aligned} P\{x \in [i, i + F(i))\} &= \displaystyle\int_i^{i+F(i)}\frac{1}{n}dx\\ &= (i + F(i) - i) \times 1/n\\ &= F(i)/n,\\ \\ P\{x \in [i + F(i), i + 1)\} &= \displaystyle\int_{i+F(i)}^{i+1}\frac{1}{n}dx\\ &= (i + 1 - (i + F(i))) \times 1/n\\ &= (1-F(i))/n \end{aligned}$

Let’s denote the set of values $j$ that satisfies $A(j) = i$ as $A^{-1}(i)$ . The generated variable $Y'$ has following probability mass function:

 $P\{Y' = i\} = F(i)/n + \sum_{j \in A^{-1}(i)}\frac{1-F(j)}{n}$

Alias method is the algorithm to construct $A$ and $F$ so that $P\{Y' = i\}$ equals to $P\{Y = i\}$ for all $i$ . Because the domain of both $A$ and $F$ are integers $0,1,\ldots,n-1$ , they can be stored in array and values can be looked up in O(1), where the space efficiency is in O(n).

In miloyip’s implementation, $A$ and $F$ are stored in std::vector<AliasItem> mAliasTable, where $A$ ’s values are stored in AliasItem::index and $F$ ’s values are AliasItem::prob.

Algorithm

Construct Steps

Initialize the set $S$ to be ${0,1,\ldots,n-1}$ and n variables $p_i$ that with values:

 $p_i = P\{Y=i\}, i \in S$

Denote the number of elements in $S$ as $\|S\|$ . We have a important invariant that

 $\sum_{i \in S}{p_i} = \|S\| / n$

At the beginning of the algorithm, the invariant holds because the sum of all probabilities must equal to 1.

The algorithm is performed using following steps.

If there is an element $i$ in set $S$ such that $p_i < 1/n$ , there must be a $j$ in set $S$ such that $p_j > 1/n$ .¹ Let $A(i) = j$ and $F(i) = p_i / (1/n) = p_i \times n$ . Remove $i$ from $S$ and subtract $n/1 - p_i$ from $p_j$ . It is easy to verify that the invariant still holds after these changes.²
Repeat step 1 until $S$ is empty or there is no more elements $i$ in $S$ that $p_i < 1/n$ . If $S$ is empty, the algorithm finishes. Otherwise for all remaining $i$ in $S$ , we must have $p_i = 1/n$ .³ Let $A(i)=i$ and $F(i)=p_i\times n=1$ for all remaining $i$ , and remove them from the set $S$ .

The algorithm finishes when $S$ becomes empty, and an element is removed only when its corresponding $A$ and $F$ has been determined, so all values of $A$ and $F$ has been generated.

In miloyip’s implementation, $p_i$ is stored in AliasItem::prob before $i$ is removed from the set. When $i$ is removed from the set, AliasItem::prob is set to $F(i)$ .

Correctness

The invariant holds at the beginning and at the end of each step, it guarantees that the algorithm can finish. It is easy to prove it using mathematical induction. So we only need to prove $P\{Y'=i\}=P\{Y=i\}$ for any $i$ , i.e.,

 $P\{Y = i\} = F(i)/n + \sum_{j \in A^{-1}(i)}\frac{1-F(j)}{n}$

Denote $p'_i$ as the value of $p_i$ when $i$ is removed from set $S$ . Check the construction steps again, we get following properties:

No $p_i$ can increase. Thus $p_i <= P\{Y=i\}$ in all steps and $p'_i <= P\{X=i\}$ .
$p_i$ decreases only when its initial value $P\{Y=i\}>1/n$ . So if $P\{Y=i\}<=1/n$ , $p_i = P\{Y=i\}$ throughout the algorithm and $p'_i=P\{Y=i\}$ .
$F(i) = p'_i \times n$
$i$ is removed only when $p_i \leq 1/n$ , i.e., $p'_i \leq 1/n$ , thus $F(i)=p'_i \times n \leq 1$ .
$A(j)$ is set to a value $i \neq j$ only if $p_i > 1/n$ (see step 1), i.e., $P\{Y=i\}>1/n$ .

Now consider value $i$ when $P\{Y=i\}<1/n$ , $P\{Y=i\}=1/n$ and $P\{Y=i\}>1/n$ .

P{Y=i} < 1/n

If $P\{Y=i\} < 1/n$ , from property 2 and property 3, $F(i) = p'_i \times n = P\{Y=i\} \times n$ .

Apparently $A^{-1}(i) = {}$ , because $A$ is either set to value $j$ where $p_j>1/n$ in step 1 or $k$ where $p_k = 1/n$ in step 2.

Thus

 $\begin{aligned} &F(i)/n + \sum_{j \in A^{-1}(i)}\frac{1-F(j)}{n}\\ =&F(i)/n\\ =&P\{Y=i\} \times n / n\\ =&P\{Y=i\} \end{aligned}$

which completes the proof.

P{Y=i} = 1/n

If $P\{Y=i\} = 1/n$ , apparently $A(i) = i$ . If there’s another value $j\neq~i$ also satisfies $A(j) = i$ , from property 4, $P\{Y=i\} > 1/n$ , conflict with the condition. So $A^{-1}(i) = {i}$

Thus

 $\begin{aligned} &F(i)/n + \sum_{j \in A^{-1}(i)}\frac{1-F(j)}{n}\\ =&F(i)/n + (1-F(i))/n\\ =&1/n \end{aligned}$

which completes the proof.

P{Y=i} > 1/n

When $P\{Y=i\} > 1/n$ , apparently i is not in $A^{-1}(i)$ .

Consider each value $j$ in set $A^{-1}(i)$ . Once $j$ is removed from $S$ , $A(j)$ is set to $i$ and $1/n - p'_j$ is subtracted from $p_i$ . Thus

 $p'_i = P\{Y=i\} - \sum_{j \in A^{-1}(i)}(1/n - p'_j)$

Then

 $\begin{aligned} &F(i)/n + \sum_{j \in A^{-1}(i)}\frac{1-F(j)}{n}\\ =&p'_i \times n / n + \sum_{j \in A^{-1}(i)}\frac{1-(p'_j \times~n)}{n}\\ =&P\{Y=i\} - \sum_{j \in A^{-1}(i)}(1/n - p'_j)\ + \sum_{j \in A^{-1}(i)}(1/n - p'_j)\\ =&P\{Y=i\} \end{aligned}$

For all $i$ , $P\{Y'=i\} = P\{Y=i\}$ , the proof completes.

Intuitive Presentation

The algorithm can be presented in intuitive meaning. The range $(0, n]$ is split into n consecutive sub ranges $(i, i + 1]$ for $i = 0, 1, \ldots, n - 1$ . The probability of $X$ falls into any range is $(i + 1 - i) \times 1/n = 1/n$ .

For $P\{Y=i\} = 1/n$ , we can allocate the whole slot $i$ to it. Let $Y=i$ when $x$ falls in $(i, i + 1]$ which has the probability $1/n$ .

If $P\{Y=i\} < 1/n$ , we can allocate the starting part $(i,i+n\times~P\{Y=i\}]$ in $(i,i+1]$ . Let $Y = i$ when $x$ falls in $(i, i + n\times P\{Y=i\}]$ , where the probability is $n\times~P\{Y=i\}\times(1/n)=P\{Y=i\}$ .

If $P\{Y=i\} > 1/n$ , we can allocate unused ranges in $(j + n\times P\{Y=j\}, j + 1]$ for any $j$ that $P\{Y=j\} < 1/n$ . However, unused range is not allowed to be split again.

See the figure below, which demonstrates how to generate $Y$ with probability mass function $n = 5$

Alias Method

$P\{Y=0\} = 0.16$
$P\{Y=1\} = 0.1$
$P\{Y=2\} = 0.32$
$P\{Y=3\} = 0.22$
$P\{Y=4\} = 0.2$

$P\{Y=4\}=1/n$ , so let $Y = 4$ only when $x$ falls in $(4, 5]$ , which probability is $(5-4)\times 0.2 = 0.2$ .

$P\{Y=0\}=0.16<0.2$ , so let $Y = 0$ only when $x$ falls in $(0,0.16\times~5]$ , i.e., $(0,0.8]$ , which probability is $(0.8-0)\times~0.2=0.16$ . $(0.8,1]$ is unused.

$P\{Y=1\}$ is the same. $(1,1.5]$ is allocated and $(1.5,2]$ is unused.

$P\{Y=2\} = 0.32 > 0.2$ , it needs ranges with total length $0.32\times~5=1.6$ . We allocate the range $(0.8, 1]$ and $(1.5, 2]$ . The remaining length $1.6-0.2-0.5=0.9<1$ , then we can allocate a part of its own slot. Finally, three ranges have been allocated, $(0.8,1]$ , $(1.5,2]$ and $(2,2.9]$ . $(2.9,3]$ is unused.

Follow the same step to handle $Y=3$ . The final allocation is depicted in $D$ . The allocation is not unique, $F$ depicts another solution.

References

If all $j$ except $i$ that $p_j \leq 1/n$ , Sum up both end of the inequalities for all $j$ and $p_i < 1/n$ , we can get $\sum_{i \in S}{p_i} < \|S\| / n$ which is conflict with the invariant.
The right side has decreased $1/n$ because $\|S\|$ has decreased 1. The left side has decreased $p_i + (n/1 - p_i) = 1/n$ , because $i$ is removed from the set and $(n/1 - p_i)$ is subtracted from $p_j$ . Thus both side decrease the same amount, the equality still holds.
Because no $p_i < 1/n$ , then $p_i \geq 1/n$ . To satisfy the invariant, no $p_i$ can be larger then $1/n$ . Thus for all $i$ in $S$ , $p_i = 1/n$ .