logo
Antony Jepson
🤔 About
✍️ Contact
21 Sep 2020
 

Notes from setting up a disconnected Debian Testing server
22 August 2020

I recently set up a new server at-home server that isn’t connected to the Internet. It’s only available on the local network and doesn’t have any connection to the outside world. It’s part of my longer term initiative to have a disconnected household.

Here’s some notes I took while setting this up.

Why?

I have servers scattered across various providers and felt a general anxiety about security. While I do feel confident using online storage solutions like AWS S3 Glacier class + GPG encryption, I feel that instance-based compute services would likely cause me a headache in the future. Storing everything locally would be faster and reduce costs as well.

Updates

After a fresh installation of Debian Testing, I briefly plugged in the Ethernet cable, set my sources.list to point to ftp.uk.debian.org, and downloaded the latest packages and security updates including vim, sudo, and screen.

For future updates, I download the weekly testing DVD image from Debian’s servers and transfer them via SSH. I’m still working on optimising this.

Each weekly testing DVD is archived in storage.

Screen suspend

As this is a laptop, the screen didn’t suspend automatically when it wasn’t in use. I appended consoleblank=60 to my GRUB_CMDLINE_LINUX_DEFAULT entry.

Containers

I successfully migrated my containers from Docker to qemu and systemd-container based deployments. I’ll detail more about this in the future. For each deployment I check that it deploys without problems in another physical machine.

Alarms as timers
13 May 2020

07:59.

It was an unusually overcast morning and the birds seemed to be reluctantly chirping, upset at the sun apparently missing its promised sun rise time.  While I was running, my alarm went off.  I realised that I was way ahead of my usual self – I had prepped breakfast, almost finished my run, and had solved a couple long standing issues at work; all while watching the clouds mope across the sky.

This made me realise – having the alarm wake you up at a certain time is all wrong.  Rather, it should be viewed as a timer by which you are ready to attack your day.  Let’s be honest to ourselves, who jolts out of bed with their hair already combed, coffee half drank, and laptop furiously churning through the results streaming in from that advanced SQL query you wrote 10 minutes ago?

Next time, I’ll try to have finished my run and already showered and dressed before the alarm goes off.

 

My principles of product management
4 March 2020

[Draft post]

Principles of product managemnet

I’ve been in product management for a while and over the course of the past 8 years I’ve learned a lot about the difficulties that aren’t apparent in the popular posts promoting product management as a career choice.

Namely,
– that the ability to connect with other humans is under-rated…
– that the development team’s well-being is just as important as your own well-being…
– and, that most customers give you bad, and non-actionable feedback…

So from these realisations and my experience, here are my 5 principles of product management.

1) Focus on the customer problem, not the solution.
Too often I see my peers around getting enthralled by some solution that seems to address every customer need. My first question to these budding PMs is: have you defined the customer problem? The second is how to did you define this? More often than not, the series of answers is: Yes, and Gut Feel.

While I do champion gut feel for making some difficult decisions I don’t think it should be ascribed to critical decisions. Rather, focusing on customer data and customer feedback is the reality when deciding on the solution and then moving onto the solution.

2) A/B testing is validation not truth.
Often I find that junior Product Managers want to focus on testing, testing, testing. From experience, the more important portion is crafting a meaningful hypothesis to test rather than getting a metric to be greater than another.

More often than not, if an experiment becomes wins by a wide margin – something is not right. Give credit to the organisation you joined – most of the low hanging fruit has already been eaten. If you have a huge win, it’s highly probably that it’s a fluke. You measured wrong, your hypothesis doesn’t actually address the customer problem, or it was a seasonal occurrence. I encourage you to take a deep look at the data across multiple facets and see if that huge win (which, by the way gets you plenty of kudos) is actually going to improve your product.

3) Failure is not optional.
As a product manager, you should be constantly questioning your own opinions and deductions on how the world works. Your experience, although great (or limited), doesn’t necessarily match how your customers perform. The only way to craft a winning path is to avoid the obstacles along the way and that means making mistakes — fast — and changing direction quickly.

4) You are the blocker
I’ve found that often the PM is the blocker to progress. Why? Because they don’t communicate their vision well enough. Tell the devs exactly what you want — and what you don’t. Work with them to a great compromise. Coordinate with design on your thoughts — and don’t rely on them to eventually iterate into what you want. Check your pulse with user research and quickly incorporate their feedback. Communicate with other product managers what you plan to do so they can adjust as required.

5) Micro pivots are as effective as major movement
Often you’ll instrument something that shows one of your earlier convictions was wrong or even dead wrong. It’s OK to make a small change that fixes that without changing your main product. Not everything needs to be a huge change with tens of other teams involved. Make a judgement call and commit to the changes that you think are necessary to improve the product and be prepared to turn off that feature flag if it doesn’t work and commit it if it does.

These are just a few of the principles that help to guide me when developing a product. Disagree? Agree? Please head over to me on Twitter @plktio or email, see my contact info, and let me know what you think. I’ll publish the best comments on my blog next week!

Building plkt.io
4 March 2019

I’m planning out my new website and homepage: plkt.io. It’s where all my posts will live on a static site that I’ll design and build myself. I’ll also be integrating services such as Gitlab, Discourse, and other interactive open source projects to make my website more inclusive. Furthermore, finances permitting, I’ll be providing a limited amount of bandwidth for sharing custom builds of various open source projects, such as Firefox, to be available for download.

This is a big step for me and the first time I focus on really building a website that truely represents me beyond my writing ability. It’s also the first time I use modern open source web stacks to build a mobile/desktop website.

I also plan to start streaming my coding sessions via Twitch and store recordings on this website. I really think interactive and public coding sessions are only going to become bigger in the future.

Stay tuned for more info and when the first draft will be pushed online.

2019 Goals and Tentatives
9 January 2019

I’m not one to dive into too much detail about my personal, soft development goals but I do have some high level technical-based objectives for this website and myself for the remainder of 2019.

– Build a better landing page for this website that makes it simple to find an article that you’re interested in reading. This website contains over ten years of content and right now all people see is the latest post. Arriving at my page from a web search is OK but doesn’t lend to much appreciation of the site’s content.
– Gather some basic metrics about the site: time spent on each page, etc., without using Google. I already get the default Apache logs but this is really just for debugging page issues. As of now I don’t get many comments on my content so there’s really no feedback loop.
– Cover some AI/ML topics – explore using CUDA acceleration. I’m fascinated by the better approximations afforded by AI/ML and want to see if I can use this to generate a revenue stream.
– Dive deeper into Docker and K8s. Both are quickly becoming the gold standard for containerisation of applications and I would like to see them continue to grow. In some ways, however, I wonder if over usage of the two leads to poor optimisation and growing long term costs for an organisation even when you consider the collaboration benefits.
– Cover basic testing, CI, and CD practices. I have to acknowledge that code and testing now go hand in hand and so would like to become more up to date in this area.
– Optimise the site display on mobile. I don’t have analytics on this but I suspect that most people land here from mobile. The theme I’m using is outdated and doesn’t display well on a low end device.
– Contribute to Firefox – I’m concerned about the homogenisation of browser backends.
– Explore more media @ home solutions like Plex. My usage of YouTube has gone down over the years as I’ve acquired more boxsets and I’d like to watch the higher quality video on my local network instead of through YouTube.
– More posts around system administration. I consider myself an amateur system administrator and can rise to the challenge of most system administration problems. Usually this involves late nights crawling through obtuse documentation and programs not conforming to said documentation but I enjoy the challenge.
– Explore using a remote PC to stream gaming content over a local network. I enjoy my games but don’t like to use an entire PC just for gaming. So, I’d like to pass the graphics card to a virtual machine that can handle gaming on demand.

Let me know if you have any suggestions on the above.

Fresh Ubuntu 18.04 Install
4 January 2019

There’s something insanely refreshing about a brand new Linux install. I recently retired my Fortnite gaming rig and turned it into an Docker/CUDA server for running AI/ML on my two NVidia GPUs. While Fortnite will certainly be missed, there’s much more utility in having a bespoke local Linux server.

Here are the typical things I do upon setting up a fresh bare-metal headless machine.

Schedule a daily wake up
It’s easy to take for granted that you have physical access to the server. But what happens if you shutdown the computer by accident or there’s a power outage? I immediately make sure I have RTC wake up enabled in the BIOS and schedule a daily wake up if the option is available.
I also schedule a daily wake up in the OS as well for added convenience. Unfortunately, if you actually wanted to keep the server off and disconnected your best bet would be to mangle the boot config until you have access again.

Set up the back up storage
All my local servers have an additional drive, either SSD or HDD, for backups. My preference is to have an ext4 mirrored RAID mounted at either /data or /srv. If I set up a *BSD instance, then I’ll opt for ZFS if I can.
Backups are not as automated as I’d like currently as I’m still fine tuning the manual process prior to automating. On Linux, I’ll do a double backup. First I create an LVM logical volume snapshot of the root partition. I then take an image of the ext4 filesystem using e2image, and place that on the additional drive. Here’s a sample invocation:

# lvs
  LV     VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root  vg -wi-ao---- <22.31g
  swap_1 aves-vg -wi-ao---- 976.00m  
# lvcreate --size 1G --snapshot --name root-snapshot-20190104 /dev/vg/root
# lvs
  LV                     VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root                  vg owi-aos--- <22.31g
  root-snapshot-20190104 aves-vg swi-a-s---   1.00g      root   0.01
  swap_1                 vg -wi-ao---- 976.00m
# e2image -r -a -p /dev/vg/root-snapshot-20190104 /srv/root-snapshot-20190104.img
# mksquashfs /srv/root-snapshot-20190104.img /srv/root-snapshot-20190104.squashfs -info

It's possible to combine the last command like so - although this seems error prone unless automated in a script.
# mktemp -d
/tmp/tmp.FM0xNQ0v4r
# mksquashfs /tmp/tmp.FM0xNQ0v4r /srv/root-snapshot-20190104.squashfs -p "root-snapshot-20190104.img f 444 root root e2image -r -a /dev/vg/root-snapshot-20190104 -"

For the second back up, I'll just tar up the file system, ignoring /dev and mounted filesystems.
# mktemp -d
/tmp/tmp.1cu7jszuVl
# mount /dev/vg/root-snapshot-20190104 /tmp/tmp.1cu7jszuVl
# echo "'root-snapshot-20190104.img': snapshot of system after initial install of Ubuntu 18.04 with OpenSSH server enabled by default" >> /srv/backup-history.log
# tar cf /srv/root-snapshot-20190104.tar --one-file-system /tmp/tmp.1cu7jszuVl
# unmount /tmp/tmp.1cu7jszuVl

And finally I'll remove the snapshot and continue as if nothing happened :).
# lvremove /dev/vg/root-snapshot-20190104

Set the hostname
IP addresses are rather annoying to remember on a DHCP home network. They change frequently and are generally not what you want to use for a custom network. There's two solutions to this, either a) reference the computers by their MAC address; or b) use multicast DNS to reference them by hostname. I always opt for the latter. On macOS, this works out the box. On Ubuntu, you need to install avahi. Normally this automatically configures your nsswitch file but in the event that's not done you'll need to change the hosts line.
$ sudo apt install libnss-mdns
## /etc/nsswitch.conf
hosts: files mdns4_minimal [NOTFOUND=return] dns

Now you can ping your server hostname.local.

Lock down the installation
Usually the only optional component I add is the OpenSSH Server. I generate keys on my local computer and transfer them over using ssh-copy-id. Then I disable root account log in and password authentication.
## /etc/ssh/sshd_config
PasswordAuthentication no
PermitRootLogin no

Get comfy
If you don't feel like you're being plugged into the matrix everytime you open the terminal, you're doing something wrong. The Terminal should feel like the defacto interface to your servers -- at least until something better comes along.

Setting up macOS Mojave
23 November 2018

I recently upgraded my 2011 Macbook Air to a 2018 Macbook Pro running macOS Mojave and I must say it runs splendidly. Here are a few things I install on a upon a new installation to become comfortable.

Graphical apps

  • Firefox is the default browser for me on macOS, despite it always “using significant energy.”
  • Scriviner is used for authoring of posts offline, which I then copy and re-format into WordPress.
  • I use Xcode extensively to write some relatively simple C programs. I also use the Command Line Tools that come with the installation, although you don’t need to install them together.
  • Screen sharing is a handy VNC client that works better than X forwarding, at least with the default installation.
  • Notes is the preferred note taking app that syncs pretty seamlessly across my phone and computer. Usually I put quick unsorted notes here and later export them to OneNote in a finalised form for reference later.
  • Microsoft Office is the de facto application suite that I’ve been using for well over a decade. It’s required for building decks, long form writing, and handling mail that needs to be shared in a business setting. If I keep the output local to me, then I’ll stick with LaTeX.
  • VLC is the default for watching videos or listening to radio streams, such as the excellent Groove Salad on Soma.FM. I’ve tried using the Radio section on iTunes but it just seems to be too buggy compared to a bespoke .pls playlist file.

Terminal apps

I use a variety of console apps – some of which aren’t available on macOS by default. Xocite.com has a pretty good tutorial for how to get started on setting up local apps on Mac.

Screen

Screen is my favourite terminal multiplexer. Here is the config I use.

# Interaction
escape ``
screen -t default 1 bash
bindkey "^[[1;5D" prev # ctrl-left
bindkey "^[[1;5C" next # ctrl-right

# Behavior
term screen-256color
startup_message off
vbell off
altscreen on
defscrollback 30000

Vim

An improved version of the Vi editor, according to the documentation. For me, it’s my primary text editor on the console. The configuration below goes on every user account. As you can tell, I like my tabs two characters wide, with spaces. I don’t use any plugins.

filetype on
filetype plugin on
filetype plugin indent on

syntax enable
set ttyfast

set tabstop=2
set shiftwidth=2
set softtabstop=2
set smarttab
set expandtab
set autoindent
set smartindent
set cursorline
set nobackup
set nowritebackup
set nocompatible
set noswapfile
set backspace=indent,eol,start

set secure
set enc=utf-8
set fenc=utf-8
set termencoding=utf-8

LaTeX

I use the excellent LaTeX for authoring of my resume and diagramming on my blog posts. I primarily use MacTeX for now but would like to switch to something that doesn’t need root privileges – essentially just a package I can extract in any directory and start using.

Keyboard shortcuts

Yes, believe it or not, knowing your keyboard shortcuts goes a long way on macOS, especially when having the laptop docked. I primarily use the window management keyboard shortcuts built-in.

  • ⌃↑: Mission Control, also bound to top mouse button
  • ⌃↓: Application window, also bound to bottom side mouse button
  • ⌃←: Move left a space
  • ⌃→: Move right a space
Understanding SHA256 Part 2
12 May 2018

This is the second in a three part series where we break down the SHA-2 algorithm. In this part, we’ll perform the compression algorithm upon the message to give us the final hash.

From before, we have the message M, shown below.

Block M^1^~1~  = 0x534E5344
Block M^1^~2~  = 0x80000000
Block M^1^~3~  = 0x80000000
[....blocks of zeroes.....]
Block M^1^~16~ = 0x00000020

Now we’ll start compressing the message to generate the hash.

Hash computation

We create a message schedule from the official documentation of 64 32-bit words, 8 32-bit temporary variables, and the final hash value of 8 32-bit words. The words of the message schedule have the label W0 to W63. We label the 8 temporary variables a through h and the hash values as H(i)0 through H(i)7 with the initial value being in H(0). These hash values will be replaced by the next stage hash value after each message block is processed. The final hash value will be H(N). Finally, we set aside two temporary words, T1 and T2.

Begin by setting the initial hash value to be the fractional square root of the first 8 primes.

H^0^ = 0x6a09e667
H^1^ = 0xbb67ae85
H^2^ = 0x3c6ef372
H^3^ = 0xa54ff53a
H^4^ = 0x510e527f
H^5^ = 0x9b05688c
H^6^ = 0x1f83d9ab
H^7^ = 0x5be0cd19
M^1^ = shown above

Then we run the following loop over each message block, i = 1 to N:

  1. Set the message schedule Wt:
    1. For t = 0 (inclusive) to t = 15 (inclusive), set Wt = M0t
    2. For t = 16 (inclusive) to t = 63 (inclusive):
      1. set Wt = σ(W(t-2), 1, 256) + W(t-7) + σ(W(t-15), 0, 256) + W~t(-16)
  2. Set the temporary variables:
    1. a = H(i-1)0
    2. b = H(i-1)1
    3. c = H(i-1)2
    4. d = H(i-1)3
    5. e = H(i-1)4
    6. f = H(i-1)5
    7. g = H(i-1)6
    8. h = H(i-1)7
  3. For t = 0 to t = 63:
    1. Set T1 = h + sum(e, 1, 256) + Ch(e, f, g) + Kt + Wt, Kt being the t-th prime cube root
    2. Set T2 = sum(a, 0, 256) + Maj(a, b, c)
    3. h = g
    4. g = f
    5. f = e
    6. e = d + T1
    7. d = c
    8. c = b
    9. b = a
    10. a = T1 + T2
  4. Compute the ith interstep hash value Hi:
    1. H(i)0 = a + H(i-1)0
    2. H(i)1 = a + H(i-1)1
    3. H(i)2 = a + H(i-1)2
    4. H(i)3 = a + H(i-1)3
    5. H(i)4 = a + H(i-1)4
    6. H(i)5 = a + H(i-1)5
    7. H(i)6 = a + H(i-1)6
    8. H(i)7 = a + H(i-1)7

This gives us a message schedule of the following.

W[0]: 0x534e5344
W[1]: 0x80000000
W[2]: 00000000
W[3]: 00000000
W[4]: 00000000
W[5]: 00000000
W[6]: 00000000
W[7]: 00000000
W[8]: 00000000
W[9]: 00000000
W[10]: 00000000
W[11]: 00000000
W[12]: 00000000
W[13]: 00000000
W[14]: 00000000
W[15]: 0x00000020

Once complete, the final hash is as follows: H^(N-1)0 || H^(N-1)1 || H^(N-1)2 || H^(N-1)3 || H^(N-1)4 || H^(N-1)5 || H^(N-1)6 || H^(N-1)7

Computing the hash with the program gives us:

Hash: d440a8ff c4ea4abe 93fedd94 b82e0d00 b814cc52 b65e6dff 924fccfb 6f7b429d

In the next and final part of this series, we’ll take the code (below) used to generate the hash, clean it up, and compare it with some existing implementations.

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <inttypes.h>

#define PRIME_NUMBERS 2, 3, 5, 7, 11, 13, 17, 19, 23, 29,  \
31, 37, 41, 43, 47, 53, 59, 61, 67, 71, \
73, 79, 83, 89, 97, 101, 103, 107, 109, 113, \
127, 131, 137, 139, 149, 151, 157, 163, 167, 173, \
179, 181, 191, 193, 197, 199, 211, 223, 227, 229, \
233, 239, 241, 251, 257, 263, 269, 271, 277, 281, \
283, 293, 307, 311

#define SHA_LENGTH 256

void print_step(char * func_name, uint32_t input, uint32_t output) {
  printf("[%s]: input: %#.8x\n", func_name, input);
  printf("[%s]: out  : %#.8x\n", func_name, output);
}

struct m_prime {
  unsigned int pos;
  uint32_t sqrt_frac, cbrt_frac;
  unsigned long prime;
  double sqrt, cbrt;
};

struct message_block {
  uint32_t block[16];
  struct message_block * next;
  uint8_t position;
};

struct message_schedule {
  uint32_t block[64];
  struct message_schedule * next;
};

struct message_block * new_message_block(void) {
  struct message_block * m = calloc(1, sizeof *m);
  m->next = NULL;
  return m;
}

void free_message_block(struct message_block * m) {
  if (m == NULL) return;
  free_message_block(m->next);
  free(m);
}

struct message_schedule * new_message_schedule(void) {
  struct message_schedule * m = calloc(1, sizeof *m);
  m->next = NULL;
  return m;
}

void free_message_schedule(struct message_schedule * m) {
  if (m == NULL) return;
  free_message_schedule(m->next);
  free(m);
}


struct hash {
  uint32_t value[8];
};

struct message_block * pad_message(char * input) {
  size_t len = 0;
  uint32_t * blk;
  uint16_t remainder;
  char ch;
  struct message_block *head, *p;
  head = new_message_block();

  p = head;

  // Determine how many message blocks we need.
  // We need [data][1 bit][zero padding][length]
  // [data] maximum length 447 bits
  // [1 bit] length = 1 bit
  // [zero padding] maximum length 447 bits
  // [length] 64 bits

  if (input == NULL) {
    head->block[0] = 1u << 31;
    return head;
  }
  
  while (*(input + len) != '\0') {
    len++; // we have one character
    blk = &p->block[p->position];
    ch = *(input + len - 1);

    switch (len % 4) {
      case 1:  *blk |= ch << 24;
              break;
      case 2:  *blk |= ch << 16;
              break;
      case 3:  *blk |= ch << 8;
              break;
      default: *blk |= ch;
              break;
    }

    if (p->position == 15 && (len % 4) == 0) { // At the end of the block
      p->next = new_message_block();
      p = p->next;
    } else { 
      if (len % 4 == 0) p->position++;
    }
  }

  // Append the one after this block.
  p->block[p->position] += (1u << (32 - 8 * (len % 4) - 1));

  // Calculate how many zeroes is needed.
  remainder = (len * 8) % 512;
  if (remainder > 447) { // Get a new block
    p->next = new_message_block();
    p = p->next;
  }

  // Convert the length to count of binary.
  len *= 8;

  p->block[14] = (uint32_t) len >> 16;
  p->block[15] = (uint32_t) len;

  return head;
}

uint64_t calculate_fractional(double floating) {
  /* IEEE-754 double-precision
   * S: 1 bit, sign
   * E: 11 bits, exponent
   * M: 53 bits, mantissa
   */

  uint64_t f, exponent, mantissa, fractional;

  f = *((uint64_t *) &(floating));
  exponent = ((f >> 13 * 4) & 0x7ff) - 0x3ff;
  mantissa = f & 0x000fffffffffffff;
  fractional = (((mantissa << exponent) & 0x000ffffffff00000)) >> 5 * 4;  /* Only last 4 bytes). */

  return (uint32_t) fractional;
}

struct m_prime * init_prime_list(unsigned int num) {
  /* Initialise variables. */
  unsigned int x;
  const unsigned int primes[]= {PRIME_NUMBERS};
  struct m_prime *p, *mp = malloc(num * sizeof *mp);

  p = mp;

  for (x = 0; x < num; x++) {
    p->prime = primes[x]; 
    p->sqrt = sqrt(primes[x]);
    p->cbrt = cbrt(primes[x]);
    p->sqrt_frac = calculate_fractional(p->sqrt);
    p->cbrt_frac = calculate_fractional(p->cbrt);

    p++;
  }

  return mp;
}

void print_prime_list(struct m_prime * p, unsigned int num) {
  unsigned int x;

  printf("| Numb | Prme |   Sqrt |   Cbrt | Frac (b10) |   Frac (b2) |\n");
  printf("|------|------|--------|--------|------------|-------------|\n");

  for (x = 0; x < num; x++) {
    printf("| %4u | %4lu | %2.4f | %2.4f | %#.8x | %#.8x  |\n", 
        x + 1,
        p->prime,
        p->sqrt,
        p->cbrt,
        p->sqrt_frac,
        p->cbrt_frac
    ); 

    p++;
  }
}

uint32_t rotate_right(uint32_t x, uint16_t n) {
  return (x >> n) | ((x << (32 - n)));
}

uint32_t hash_sigma_0(uint32_t x) {
  return rotate_right(x, 7) ^ rotate_right(x, 18) ^ (x >> 3);
}

uint32_t hash_sigma_1(uint32_t x) {
  return rotate_right(x, 17) ^ rotate_right(x, 19) ^ (x >> 10);
}

uint32_t hash_sum_0(uint32_t x) {
  return rotate_right(x, 2) ^ rotate_right(x, 13) ^ rotate_right(x, 22);

}
uint32_t hash_sum_1(uint32_t x) {
  return rotate_right(x, 6) ^ rotate_right(x, 11) ^ rotate_right(x, 25);
}

uint32_t hash_ch(uint32_t x, uint32_t y, uint32_t z) {
  return (x & y) ^ (~x & z);
}

uint32_t hash_maj(uint32_t x, uint32_t y, uint32_t z) {
  return (x & y) ^ (x & z) ^ (y & z);

}
 
struct message_schedule * compute_message_schedule(struct message_block * mb) {
  uint8_t i;
  struct message_schedule *ms;

  if (mb == NULL)
    return NULL;

  ms = new_message_schedule();
  for (i = 0; i < 16; i++) {
    ms->block[i] = mb->block[i];
  }

  for (i = 16; i < 64; i++) {
    ms->block[i] = hash_sigma_1(ms->block[i - 2]) + ms->block[i - 7] + hash_sigma_0(ms->block[i - 15]) + ms->block[i - 16];
  }
  ms->next = compute_message_schedule(mb->next);

  return ms;
}

struct hash * perform_hash(struct hash *hash, struct message_schedule * m, struct m_prime * p) {
  if (m == NULL) return hash;

  uint32_t a, b, c, d, e, f, g, h, tx, ty;
  uint8_t i;
  a = hash->value[0];
  b = hash->value[1];
  c = hash->value[2];
  d = hash->value[3];
  e = hash->value[4];
  f = hash->value[5];
  g = hash->value[6];
  h = hash->value[7];

  for (i = 0; i < 64; i++) {
    tx = h + hash_sum_1(e) + hash_ch(e, f, g) + (p + i)->cbrt_frac + m->block[i];
    ty = hash_sum_0(a) + hash_maj(a, b, c);
    h = g;
    g = f;
    f = e;
    e = d + tx;
    d = c;
    c = b;
    b = a;
    a = tx + ty;

    // printf("%.2u: %#.8x %#.8x %#.8x %#.8x %#.8x %#.8x %#.8x %#.8x\n", i, a, b, c, d, e, f, g, h);
  }

  hash->value[0] += a;
  hash->value[1] += b;
  hash->value[2] += c;
  hash->value[3] += d;
  hash->value[4] += e;
  hash->value[5] += f;
  hash->value[6] += g;
  hash->value[7] += h;

  return perform_hash(hash, m->next, p);
} 

int main(int argc, char *argv[]) {
    struct m_prime * p;
    struct message_block *h;
    struct message_schedule *s;
    struct hash * hash = calloc(1, sizeof *hash);
    unsigned int i, size = 64;

    p = init_prime_list(size); 

    if (argc == 1) {
    h = pad_message(NULL);
    } else {
    h = pad_message(argv[1]);
    }

    for (i = 0; i < 8; i++)
      hash->value[i] = (p + i)->sqrt_frac;

    s = compute_message_schedule(h);

    hash = perform_hash(hash, s, p);

    printf("Hash: ");
    for (i = 0; i < 8; i++)
      printf("%8x ", hash->value[i]);
    printf("\n");

    free_message_block(h);
    free(p);
}
Understanding SHA256 Part 1
11 May 2018
This is the first in a three part series of articles that break down the SHA-2 algorithm. In part one, we’ll go over the initial definitions that are integral to understanding the algorithm and write sample code in C. In part two, we’ll compute the hash given the initial values we generated in part one. And then in part three, we’ll take another look at the program, comparing it against the defacto implementations and see how we can optimise it to run faster.

SHA-2

SHA-2 (Secure Hash Algorithm 2) is used trillions of times a day to compute cryptographic hashes and is essential for the stability of everything on the web. That being said I’m not exactly sure how it works and thought it would be a good exercise to sit down and break it into digestible chunks.

My reference is the PDF published on the National Institute of Standards and Technology site here.

Here are a few concepts that need to be internalised before moving forward.

Definitions

One way function
A function that is easy to compute but hard to reverse.
Ability to process arbitrary length inputs
A hash function can split up messages into smaller chunks and then operate on them sequentially.
Merkle-Damgard construction
A way to build a cryptographic hashing function that retains the collision resistant properties of the hashing function.
Time to brute-force a hash function
For a hash function where L is the bits per digest, then finding a matching message will take 2 raised to the L evaluations. Known as a pre-image attack.
Avalanche effect
A simple change in the output modifies the output drastically.
Modular arithmetic
Wrap around addition — used when telling time (modulo 12).

Functions used

The following functions operate on 32-bit words.

  • RIGHT (n) ROTATE = (x right shift n) OR (x left shift (w – n)), x = w bit word, n is integer between 0 (inclusive) and w
  • RIGHT (n) SHIFT = x right shift n, x = w bit word, n integer between 0 (inclusive) and w
  • Addition is modulo 2^32.
  • (1): Ch(x, y, z) = (x AND y) XOR (NOT x AND z)
  • (2): Maj(x, y, z) = (x AND y) XOR (x AND z) XOR (y AND z)
  • (3): Sum(x, 0, 256) = RIGHT (2) ROTATE (x) XOR RIGHT (13) ROTATE (x) XOR RIGHT (22) ROTATE (x)
  • (4): Sum(x, 1, 256) = RIGHT (6) ROTATE (x) XOR RIGHT (11) ROTATE (x) XOR RIGHT (25) ROTATE (x)
  • (5): σ(x, 0, 256) = RIGHT (7) ROTATE (x) XOR RIGHT (18) ROTATE (x) RIGHT (3) SHIFT (x) ROTATE (x)
  • (6): σ(x, 1, 256) = RIGHT (2) ROTATE (x) XOR RIGHT (13) ROTATE (x) XOR RIGHT (10) SHIFT (x)

Initialisation

First, we begin by calculating all the initial values. There are two sets: (1) the fractional portions of the square roots of the first 8 primes; and (2) the fractional portions of the cube roots of the first 64 primes. I have generated them here.

Preparing the message

Next, we prepare the message for processing. For this tutorial, I’ll be using “SNSD” as the initial message. This message has a length of 20 bits. We append a single ‘1’ bit to the end (step 1) — this specifies the end of the message in padded message. Then we add (step 2) k zero bits where k is the smallest number that makes the following equation true: l + 1 + k = 448 mod 512. Solving for k in this message gives us k = 448 – (20 + 1) = 429 zero bits. After that, we add the message length in the last 64 bit block (step 3).

S N S D 1 bit 429 zero bits Length bits
0x53 0x4E 0x53 0x44 0x1 429 * 0x0 0x00000000000014

Reading in the message

Next, we split the message into blocks so we can process it. This step is simple for this short example. The code at the end of the post handles more difficult lengths of messages. We’ll reference each N block of 512 as MN and each 32-bit word (W) within the block as MWN. Therefore, for this message we have:

  • Block M11 = 0x534E5344
  • Block M12 = 0x80000000
  • Block M13 = 0x00000000
  • [….blocks of zeroes…..]
  • Block M116 = 0x00000020

Putting it all together

The following (hastily put together, if I might add) C program will calculate the initial values that I showed earlier and the padded message. Look for part 2 where I will cover the hashing function.

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <inttypes.h>

#define PRIME_NUMBERS 2, 3, 5, 7, 11, 13, 17, 19, 23, 29,  \
31, 37, 41, 43, 47, 53, 59, 61, 67, 71, \
73, 79, 83, 89, 97, 101, 103, 107, 109, 113, \
127, 131, 137, 139, 149, 151, 157, 163, 167, 173, \
179, 181, 191, 193, 197, 199, 211, 223, 227, 229, \
233, 239, 241, 251, 257, 263, 269, 271, 277, 281, \
283, 293, 307, 311

struct m_prime {
  unsigned int pos;
  uint64_t sqrt_frac, cbrt_frac;
  unsigned long prime;
  double sqrt, cbrt;
};

struct message_block {
  uint32_t block[16];
  struct message_block * next;
  uint8_t position;
};

struct message_block * new_message_block(void) {
  struct message_block * m = calloc(1, sizeof *m);
  m->next = NULL;
  return m;
}

void free_message_block(struct message_block * m) {
  if (m == NULL) return;
  free_message_block(m->next);
  free(m);
}

struct message_block * pad_message(char * input) {
  size_t len = 0;
  uint32_t * blk;
  uint16_t remainder;
  char ch;
  struct message_block *head, *p;
  head = new_message_block();

  p = head;

  // Determine how many message blocks we need.
  // We need [data][1 bit][zero padding][length]
  // [data] maximum length 447 bits
  // [1 bit] length = 1 bit
  // [zero padding] maximum length 447 bits
  // [length] 64 bits

  
  while (*(input + len) != '\0') {
    len++; // we have one character
    blk = &p->block[p->position];
    ch = *(input + len - 1);

    switch (len % 4) {
      case 1:  *blk |= ch << 24;
              break;
      case 2:  *blk |= ch << 16;
              break;
      case 3:  *blk |= ch << 8;
              break;
      default: *blk |= ch;
              break;
    }

    if (p->position == 15 && (len % 4) == 0) { // At the end of the block
      p->next = new_message_block();
      p = p->next;
    } else { 
      if (len % 4 == 0) p->position++;
    }
  }

  // Append the one after this block.
  p->block[p->position] += (1u << (32 - 8 * (len % 4) - 1));

  // Calculate how many zeroes is needed.
  remainder = (len * 8) % 512;
  if (remainder > 447) { // Get a new block
    p->next = new_message_block();
    p = p->next;
  }

  p->block[14] = (uint32_t) len >> 16;
  p->block[15] = (uint32_t) len;

  return head;
}

uint64_t calculate_fractional(double floating) {
  /* IEEE-754 double-precision
   * S: 1 bit, sign
   * E: 11 bits, exponent
   * M: 53 bits, mantissa
   */

  uint64_t f, exponent, mantissa, fractional;

  f = *((uint64_t *) &(floating));
  exponent = ((f >> 13 * 4) & 0x7ff) - 0x3ff;
  mantissa = f & 0x000fffffffffffff;
  fractional = (((mantissa << exponent) & 0x000ffffffff00000)) >> 5 * 4;  /* Only last 4 bytes). */

  return fractional;
}

struct m_prime * init_prime_list(unsigned int num) {
  /* Initialise variables. */
  unsigned int x;
  const unsigned int primes[]= {PRIME_NUMBERS};
  struct m_prime *p, *mp = malloc(num * sizeof *mp);

  p = mp;

  for (x = 0; x < num; x++) {
    p->prime = primes[x]; 
    p->sqrt = sqrt(primes[x]);
    p->cbrt = cbrt(primes[x]);
    p->sqrt_frac = calculate_fractional(p->sqrt);
    p->cbrt_frac = calculate_fractional(p->cbrt);

    p++;
  }

  return mp;
}

void print_prime_list(struct m_prime * p, unsigned int num) {
  unsigned int x;

  printf("| Numb | Prme |   Sqrt |   Cbrt | Frac (b10) |   Frac (b2) |\n");
  printf("|------|------|--------|--------|------------|-------------|\n");

  for (x = 0; x < num; x++) {
    printf("| %4u | %4lu | %2.4f | %2.4f | %#.8lx | %#.8lx  |\n", 
        x + 1,
        p->prime,
        p->sqrt,
        p->cbrt,
        p->sqrt_frac,
        p->cbrt_frac
    ); 

    p++;
  }
}

int main(int argc, char *argv[]) {
  if (argc > 1) {
    struct m_prime * p;
    struct message_block *b, *h;
    unsigned int i, size = 64,  block_num = 0;

    p = init_prime_list(size); 
    print_prime_list(p, size);

    h = pad_message(argv[1]);
    b = h;

    do {
      for(i = 0; i < 16; i++) {
        printf("Block %2u.%2u: %8x\n", block_num, i + 1, b->block[i]);
      }
      b = b->next;
      block_num++;
    } while (b != NULL);

    free_message_block(h);
    free(p);
  } else {
    printf("Please enter a string.\n");
    return -1;
  }
}
iOS 9 Public Beta 1 Changes
20 July 2015

Apple has introduced a number of positive changes with iOS 9. This post details them the items I have found through ad-hoc usage.

– Mail app has icons when swipe is shown
– Selecting and deselecting text now has animations
– Passbook has be renamed to Wallet
– Swiping down from the home screen has more searchable features; equivalent to swiping left on the home screen
– Low battery mode
– Double-tap home button shows new switching mode
– Settings app now has search
– Apple Music is now default on US builds
– Maps has transit directions for some markets
– Context menus are now more Palm-esque, as in, do not extend across the screen
– More sharing options
– Notes supports additional media
– Recommended apps to use now shows up reliably on the lock screen

Find any other changes? Add comments below.

Built with Wordpress and Vim
© 2008 to 2020 Antony Jepson