Like github, I have two types of git repositories. Public repositories; which show up on gitweb and git-daemon and etc., that everyone can access; and private repositories, which contain my bank details.

My conf file consists of:

A set of user groups: While gitolite supports multiple keys for one user, I prefer to treat my various machines as separate users, for reasons that’ll become apparent later.
@faux = admin fauxanoia fauxhoki fauxtak
@trust = @faux alice
@semi = fauxcodd fauxwilf bob


A set of repositories, both public and private:
@pubrepo = canslations
@pubrepo = coke
@pubrepo = cpptracer
...
@privrepo = bank-details
@privrepo = alices-bank-details


Descriptions for all the public repositories, so they show up in gitweb:
repo coke
     coke = "Coke prices website"

repo cpptracer
     cpptracer = "aj's cppraytracer, now with g++ support"


And permissions:
repo @pubrepo
     RW+ = @trust
     RW  = @semi
     R   = @all daemon gitweb
     config core.sharedRepository = 0664

repo @privrepo
     RW+ = @trust


This allows trusted keys to do anything, and semi-trusted keys (i.e. ones on machines where there are other people with root) to only append data (i.e. they can’t destroy anything, and can’t make any un-auditable changes).

Next, to protect against non-root users on the host itself, I have $REPO_UMASK = 0027; in my .gitolite.rc. This makes the repositories themselves inaccessible to other users. However, gitweb needs to be able to read public repositories; the above config core.sharedRepository = 0664 does this.

This leaves only /var/lib/gitolite/projects.list (which is necessary as non-git users can’t ls /var/lib/gitolite/repositories/, so gitweb can’t discover the project list itself), and repositories/**/description, again for gitweb.

For this, I have a gitolite-admin.git/hooks/post-update.secondary of:

#!/bin/sh
chmod a+r /var/lib/gitolite/projects.list
find /var/lib/gitolite -name description -exec chmod a+r {} +


Now, gitweb can display public projects fine, and local users can’t discover or steal private repositories.

I’m going to make up some terms:

• Symbol: A group of related lines representing what English people would call a letter
• Glyph: A group of related lines that might be stand-alone, or might be combined to make a symbol

And use some existing, well defined terms. If you use one of these wrongly, people will get hurt:

• Code point: A number between 0 and ~1.1 million that uniquely identifies a glyph. They have numbers, written like U+0000, and names,
• Encoding: A way of converting information to and from a stream of bytes,
• Byte: The basic unit of storage on basically every computer; an octet of 8 bits. This has nothing to do with letters, or characters, or… etc.

Let’s start at the top:

• Here’s an grave lower case ‘a’: à. This is a symbol.
  • It could be represented by a single glyph, the code point numbered “U+00E0″ and named “Latin Small Letter A With Grave”, like above,
  • It could be represented by two glyphs, like: à.
    • This looks identical (unless your browser can’t cope, which is plausible), but is actually two glyphs; an ‘a’ (U+0061: “Latin Small Letter A”) followed by a U+0300: “Combining Grave Accent”. These two glyphs combine to make an identical symbol.
    • This is, of course, pointless in this case, but there are many symbols that can only be made with combining characters. Normalisation is the process of simplifying these cases.
    • Don’t believe me? Good. Not believing what you see is an important stage of debugging. Copy the above into Notepad, or any other Unicode-safe editor, and press backspace. It’ll remove just the accent, and leave you with a plain ‘a’. Do this with the first à and it’ll delete the entire thing, leaving nothing. See? Different.
• So, let’s assume we’re going with the complex representation of the symbol, the two code points: U+0061 followed by U+0300. We want to write them to any kind of storage, be it a file, or a network, or etc. We need to convert them to bytes: Encoding time.
  • Encodings generate “some” bytes to represent a code point. It can be anywhere between zero and infinity. There’s really no way to tell. Common encodings, however, will generate between one and six bytes per code point. Basically everyone uses one of the following three encodings:
  • UTF-8: generates between one and six bytes, depending on the number of the code point. Low-numbered code-points use less bytes, and are common in English and European languages. Other languages will generally get longer byte sequences. Common around the Internet and on Linux-style systems.
  • UTF-16: generates either two or four bytes per code point. The vast majority of all real languages available today fit in two bytes. Common in Windows, Java and related heavy APIs.
  • I have no idea what I’m doing: Anyone using anything else is probably doing so by mistake. The most common example of this is ISO-8859-*, which means you don’t care about non-Western-European people, i.e. 80% of the people in the world. These generate one byte for every code point, i.e. junk for everything except ~250 selected code points.
• Let’s look at UTF-8.
  • First code point: U+0061. This happens to be below U+007F, so is encoded in a single byte in UTF-8. This happens to align with low-ASCII, a really old encoding that’s a subset of ISO-8859. The single byte is 0×61, 0b0110001. Note that the first bit is ’0′.
  • Second code point: U+0300. This is not below U+007F, so goes through the normal UTF-8 encoding process. In one sentence, UTF-8 puts the number of bytes needed in the first byte, and starts every other byte with the bits “10″. In this case, we need two bytes, which are 0xCC, 0×80; 0b11001100, 0b1000000. Note how the first byte starts with “110″, indicating that there are two bytes (two ones, followed by a zero), and the second byte starts with “10″.
  • Note: All valid UTF-8 data can easily be validated and detected; if a byte has it’s left-most bit set to ’1′, it must be part of a sequence. No exceptions.
  • Consider

    $ xxd -c1 -b some.txt | grep -1 ': 1'
    0000046: 01100001  a
    0000047: 11001100  .
    0000048: 10000000  .
    

    , which shows the byte patterns outlined: The ‘a’ with the leading 0, then the two bytes of the combining character. xxd is the only tool you should trust when diagnosing character encoding issues. Everything else will try and harm you, especially your text editor and terminal.

• UTF-16 is much easier to recognise and much harder to confuse with other things as, for most Western text (including XML and..), it’ll be over 40% nulls (0×00, 0b00000000).
• Now you’ve done your conversion, you can write the bytes, to your file or network, and ensure that whoever is on the other end has enough information to work out what format the bytes are in. If you don’t tell them, they’ll have to guess, and will probably get it wrong.

In summary:

• Have some data, in bytes? Don’t pretend it’s text, even if it looks like it is; find out or work out what encoding it’s in and convert it into something you can process first. It’s easy to detect if it’s UTF-8, UTF-16 or if you have a serious problem.
• Have some textual information from somewhere? Find an appropriate encoding, preferably UTF-8 or UTF-16, to use before you send it anywhere. Don’t trust your platform or language, it’ll probably do it wrong.
• Can’t work out what’s in a file? Run it through xxd and look for nulls and bytes starting with ’1′. This’ll quickly tell you if it’s UTF-16, UTF-8 or effectively corrupt.

Hopefully that’s enough information for you to know what you don’t know.

For more, try Joel on Software: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).

This is a fork of Barry Haanstra’s PuTTY Tray, which is abandoned.

Main advantages:

• Now built against PuTTY 0.61, getting features like Windows 7 Jumplist and Aero support, and four years of core PuTTY development
• Ctrl+mousewheel zoom support
• URL detection works on URLs ending with close-brackets
• Much easier to continue development of, build script generator works and source, issue and pull-request tracking provided by github.

Please raise a bug if you have any problems or requests!

• Perl: 1.01 seconds
• Python: 2.04 seconds
• Java, including javac: 0.76 seconds
• Java, including ecj: 0.61 seconds
• Java, including javac, with -Xint: 1.01 seconds.
• Java, including javac, with -Xint on the compiler too: 1.16 seconds.

• -Xint disables practically all optimisations that Java offers, forcing the JVM into interpretation mode, so it’ll operate much like perl and python do.
• ecj is Eclipse’s compiler for Java, a faster and cleaner implementation of javac that can run standalone.

i.e. including compilation time on an entirely unoptimised compiler, Java is still twice the speed of Python.

(This isn’t really interesting or surprising to me, but the question comes up often enough that I’d like to have these here to link WRONG people to.)

Testcases:

time (N=10000000; printf "class A { public static void main(String... arg) { long j = 0; for(long i = 0; i < $N; ++i) { j += i; } System.out.println(j); } }" > A.java && javac A.java && java A)
time (N=10000000; printf "class A { public static void main(String... arg) { long j = 0; for(long i = 0; i < $N; ++i) { j += i; } System.out.println(j); } }" > A.java && javac A.java && java -Xint A)
time (N=10000000; printf "class A { public static void main(String... arg) { long j = 0; for(long i = 0; i < $N; ++i) { j += i; } System.out.println(j); } }" > A.java && javac -J-Xint A.java && java -Xint A)
time (N=10000000; printf "class A { public static void main(String... arg) { long j = 0; for(long i = 0; i < $N; ++i) { j += i; } System.out.println(j); } }" > A.java && java -jar ecj.jar -source 1.5 A.java && java A)
time (N=10000000; printf "j = 0\nfor i in range(1,$N):\n\tj = j + i\nprint j" | python -)
time (N=10000000; printf 'my $j = 0; for ($i = 0; $i < '$N'; ++$i){ $j += $i; } print $j' | perl -w)


Versions, from Debian stable:

Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
perl, v5.10.1
OpenJDK Runtime Environment (IcedTea6 1.8.7) (6b18-1.8.7-2~squeeze1), OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)
ecj 3.5.1


It, however, leaks memory. A lot of memory; about 1kb/second. Noting that it’s running all the time, and not important, this is rather inconvenient.

FTFY. Can’t redistribute a patched “binary” as the original is not redistributable.

My personal favorite situation is when you get to:
Exception in thread "main" java.lang.NullPointerException
    at com.goeswhere.dmnp.linenos.B.foo(B.java:13)

..and, line 13 is:

  System.out.println(first.substring(1) + second.toUpperCase() + third.toLowerCase());

Basically, the end of any happiness.

LineNos can fix this:

$ java -Xbootclasspath/p:linenos.jar -javaagent:linenos.jar=com/goeswhere com.goeswhere.dmnp.linenos.B
Exception in thread "main" java.lang.NullPointerException
    at com.goeswhere.dmnp.linenos.B.foo(B.java:13), attempting to invoke toUpperCase #4
    at com.goeswhere.dmnp.linenos.B.run(B.java:9)
    at com.goeswhere.dmnp.linenos.B.main(B.java:5), attempting to invoke run #2

This is implemented entirely as a Java Agent; it requires no VM modifications and is portable anywhere that supports instrumenters transforming classes (i.e. everywhere that matters).

It works by adding extra line numbers for each call on a line. Currently it adds call-number*1000 to the line number, so that it’s debuggable and easier to do in two phases, but this makes it much less efficient.

i.e., assuming Java allowed labels for line-numbers, it does:

13:
8013: System.out.println(
1013: [secret StringBuilder construction (used to implement String concatenation)]
2013: first.substring(1)
3013: +
4013: second.toUpperCase()
5013: +
6013: third.toLowerCase()
7013: [secret StringBuilder#toString()]);

Thus, the real stacktrace looks like:
Exception in thread "main" java.lang.NullPointerException
    at com.goeswhere.dmnp.linenos.B.foo(B.java:4013)
    at com.goeswhere.dmnp.linenos.B.run(B.java:9)
...

It additionally overrides StackTraceElement#toString() to decompile the named class and report the invocation that’s on that line, i.e. lookup line 4013 in the above bytecode, and report that it’s an invocation of toUpperCase, thus giving the intended result.

It has no run-time performance penalty beyond the extra time to load classes, and the increase in the size of line-number table (actually, I have no idea how this affects performance, but I can’t imagine it’s much). Printing a stacktrace is slower, however (although, it probably wouldn’t be much slower in a real implementation). Also, it’s a straw-man, so leaks memory, but this isn’t an important part of the implementation.

A much better implementation would be to store the [Class<>, line-number] -> hint mapping instead of the whole file and doing decompilation; or to replace the entire line-number table with a bytecode number table (i.e. 1->1, 2->2, 3->..), and do it all at print-time. Patches welcome.

In summary: Dear Oracle, please make the JVM do this by default. Lots of Love, Faux.

The year is 1516. The Royal Mail has just been founded.

You’re an enterprising goat farmer. You want to service requests for goats by mail. You have multiple paddocks. Paddocks contain multiple goats.

An order comes in for some goats. You spring into action!

Firstly, you need to select some goats to send out.

For this, you’ve designed a tool called the Goat Inventory Tracker. It deals with all the management aspects, so you can do what you love most; breeding goats.

You decree, Goat Inventory Tracker, Allow Me To Pick From My Goats!

The Goat Inventory Tracker shows you each of your goats across all of your paddocks, and asks you if you would like to include it. If you do, it takes the goat and places it in a pending crate.

When you’re done selecting goats, you can look into the pending crate and check everything is what you want. If it helps, you can preview your shipping log as if you had no outstanding goats.

This is important, as you’re required to maintain various arbitary constraints by the local land baron.

Now that you’re happy, you can decree Goat Inventory Tracker, Seal This Box!

This will, obviously, seal the box of goats, and place it near the post box with a shipping note of your choice.

You repeat this procedure for a couple of days. During this time, loads of boxes pile up near the mail point.

You don’t like mess.

You decree, Goat Inventory Tracker, Allow Me To Rearrange My Boxes!

You are then able to reorder, change the shipping note of, merge, and completely change or split your boxes.

Once you’re happy with your outstanding orders, you can decree Goat Inventory Tracker, Summon The Postman!

The postman will arrive, and will then check he can load your boxes into his wagon. Due to safety concerns, the postman isn’t allowed to pile your boxes on top of other people’s boxes, or allowed to re-order other people’s boxes. This means that if he has any boxes in his van already, you’ll need to do some work first.

No worry; you just decree Goat Inventory Tracker, Pile My Boxes Atop These Lowly Boxes! It will.

Sometimes your boxes won’t fit nicely on top of the other people’s, at which point the Goat Inventory Tracker will panic, and you’ll need to reshape your boxes. Remember that the Goat Inventory Tracker has many tools for dealing with boxes, as explained above.

Once you’re done piling your boxes, the postman will be happy and will leave with your boxes, and you have entered them into your account book.

You’ve made some money, you’ve made your customers happy, and you’ve done basically no work. Excellent. Back to breeding goats.

All are in the tinies v001 archive, in both x64 and legacy format. Symbols, source and signatures are available. No installer is available; just drop them into your All Programs -> Startup folder.

shiftfocus.exe adds an extra set of hotkeys for focusing windows. Ctrl+win+arrow focuses the window to that side of the current window.

This makes the most sense with Aero’s Snap turned on. If you have two windows “half-maximised” on a screen (i.e. one has been win+left‘d, and the other win+right‘d), then you can switch between them using ctrl+win+left and ctrl+win+right.

topkey.exe adds win+w to toggle a windows’ always-on-top flag, and win+return to create a new command prompt “in the current directory”. (This works for Explorer windows, and things that have the directory at the start of the title, i.e. Notepad++.)

mousex.exe allows you to use an xbox360 controller as a mouse. Different analogue sticks are different sensitivity. A/B for left/right click. Shoulder analogue controls for the scrollwheel.

And, for more niche users:

powerstatustray.exe shows which drives are spun-up, and notifies you when a drive spins up or down. (Yes, actually, this one was released before.)

keydump.exe shows what you can keylog by binding globally to DirectInput. Most keylogging preventers/detectors completely ignore this, and/or only work through blacklisting, which is laughably pointless. It doesn’t bother translating numbers into keycaps, but it’s obvious whether it’s working and whether it’s been detected.

Others, to date:

• aukiller: Legacy XP application.
• foobar2000-loader.exe: Demo of pre-loading a dll into an application via the debug api.
• keytoputty.exe: Take input and send it to a running instance of putty, i.e. to allow input during full-screen applications.
• loaddlls.exe just calls LoadLibrary on all it’s arguments.
• noelev.exe: Legacy implementation of setting __COMPAT_LAYER=RunAsInvoker.
• quickkey.exe: Legacy XP application.
• unrequireadmin.exe: An even less healthy implementation of noelev.exe.

TrickyTrucks is okay fun single-player, but what really makes it fun is the competition. For this, it has built in scoreboards, per track. Attempting to beat certain people’s times on tracks is the fun.

What it lacks is a cross-track scoreboard, i.e. some kind of championship, and/or notifications of people beating your scores. Even Audiosurf, one of the… most entertainingly engineered indie games recently, got this right.

I implemented one.

After some initial beta testing (and ensuring I was near the top of the championship), I messaged the TrickyTrucks author with the source of the scraper and of the web interface, asking for permission to link to the website on the official forum, so others could join us in competing for the championship title.

An aside, on licensing: Both of these components were released under the BSD. That allows anyone, including the TrickyTrucks author, to use the code for any purpose, including incorporating it into his official website. The component split was done such that there was a neat interface for him to implement on a non-scraper backend. The best result for me would be for there to be an official API and an officially hosted version of the site, such that I never had to do anything ever again to continue appreciating it.

An aside, on development costs: Reverse engineering binary protocols is a nightmare. Especially so with only access to a read client (with no source). Especially so when there’s no way to get the server to return things consistently or with user-specified plaintext. Especially when the protocol has (what you believe to be) NIH compression. Don’t ever, ever try and pay someone to do this unless they really, really want to.

He replied that this would be fine, but only if I removed the link from the website to the source. I, grudgingly (given it was already in the wild), did so and posted on the forum.

His response? Delete the post, and change the server to have some additional, weak protection against 3rd-party clients.

What. The. Hell.

(Eventually he sent me details of an API to use, but I’d lost interest by then.)

This post brought to you by libspotify being incompatible with most 3rd party DLLs, probably due to the copy protection on their DLL. Copy protection. On something that requires a paid account, verified against their server. Please tell them what you think about this on my GetSatisfaction thread (apparently this is what passes as a bug tracker these days).

This prompted me to waste ALL WEEKEND porting foo_input_spotify to libdespotify. foo_input_spotify will increase the value of their product. Why are they making my life miserable? Perhaps it’s unintentional. Time will tell.

Given a repo of:

My proposals for alternatives to this simple workflow are as follows. These all result in the same order of code going into the master branch, but have different histories. (Actually, I think there’s still mistakes in there but I’m tired of staring at the horrible procedural script that generates it, so it’ll do).

1. A flat history, made by rebasing everything on top of master instead of merging:

2. Only merges on master, giving you the illusion of a neat history…

…but, underneath, loads of ugly information available:

3. A hybrid, supercommits, whereby you keep a flat history but you maintain where branches were:

…or, with the history information visible:

Thoughts:

Serious concurrent projects like git itself use the ‘only merges on master’ approach. I strongly agree that they shouldn’t be flattening the history; it’s nice to be able to see groups of patches as they go into master, and to navigate the history of “feature commits”, instead of the history of “developer changes”.

The flat model seems to work much more like how I think about software development:

• You start working on something.
• You do your normal develop, commit, developer-test, commit cycle.
• Master moves on a bit while you’re messing around. The fact that you happened to start developing before a specific commit on master, instead of just after it (so it would be the root of your branch) is reasonably irrelevant; you may as well move the branch-root (or branch point) up to the top of master.
• Additionally, if you do a real merge, you need to test your changes. This leads to more developer testing, and possibly more commits.
• What do you do with these commits? Assuming your merge with master is still local you can fix it (this is git, you can fix anything), but it’s inconvenient.
• When you’ve rebased, you can continue committing on your rebased branch like normal, with confidence that when you merge it’ll all be fine (as it’ll be a fast-forward merge).

‘Supercommits’ is a hybrid of these two; you can use the rebase-onto-master workflow from ‘flat’, but you can logically group your set of commits into a… family? I like this idea, but haven’t really implemented it in practice so can’t really comment.

(Apologies for screenshots of text; I’m lazy, ansifilter was NOT WORKING and it’s prettier than gitk.)