Abstract: A modular building system formed of blocks supported around a loadbearing pipe frame. The blocks include half cylinder protruding portions located at opposing ends of the blocks wherein the protruding portions of adjacent blocks form cylindrical fastener portions. Horizontally adjacent blocks form courses, and horizontal courses of blocks are stacked to form walls. The fastener formed by the protruding portions of a lower course extend through an aperture in the lower portion of blocks forming an upper course, and a retainer member is engaged with the fastener formed by the protruding portions to thereby lock an upper course block to two lower adjacent blocks. The retainer member is formed with a through hole for receiving a vertical pipe member in sliding engagement wherein the vertical pipe members carry vertical loads without substantially transmitting vertical loads to the blocks.