Threadsafe counters markdown files.

readme.md

Thread-Safe Counters

Provides an unlimited set of named, atomicly incrementable/decrementable, counters using either the blindingly fast Redis deamon or a slower but more easily deployed filesystem implementation.

Rationale

Q: Why would you need such a beastie?

A: This repository solves a problem that can arise where a unique ID needs to be determined before data is written to the database, such as using an incremental number as the page name in ProcessWire before the page is saved. It also provides an implementation of escrow purchases where there could be a race between competing buyers to secure a limited qty of some resource or product. For more information about this, please read escrow.md.

If two or more processor threads attempt to run the same routine at the same time (two or more customers raising a bug-track issue simultaneously for example), there is the possibility that they would be assigned the same next ID before the page was saved and one would result in an error (at best).

For a discussion of the technical aspects and gotcha's that lead to the development of Thread-safe counters, please read rationale.md

Advantages

Thread-safe counters are...

• Fast: No need for round-trips to the DB to calculate or recalculate your id values.
• Portable: Works on PHP across OS platforms.
• Thread-safe: Guarantees a unique next value on every call.
• Transportable: The counter values can be moved along with your site if you need to migrate it.
• Flexible: Can increment or decrement by 1 or any value you like.
• Almost Unlimited: Use as many named counters as your application needs.
• Simple to use: As simple as calling next('your_counter_name')!

Implementations

Two implementations are provided. The first is dependency free and uses PHP's built-in flock() function in blocking mode to implement a mutually exclusive section of code. The second uses Redis and its inherent single threading to provide the needed mutual exclusion.

Repository Files

\
|- readme.md                      // This file.
|- rationale.md                   // Documentation.
|- escrow.md                      // Walks you through the escrow model supported by this code.
|
|- ThreadsafeCountersBase.php     // Abstract base class implementing common methods.
|- ThreadsafeCountersFile.php     // The filesystem-based implementation.
|- ThreadsafeCountersRedis.php    // The Redis-based implementation.
|
|- ThreadsafeFileCounters.module  // ProcessWire module that uses the ThreadsafeCountersFile.php.
|- ThreadsafeRedisCounters.module // ProcessWire module that uses the ThreadsafeCountersRedis.php.

Using The File-Based Implemetation

Basic increment of a named counter...

require_once "ThreadsafeCountersBase.php";
require_once "ThreadsafeCountersFile.php";
$unique_transaction_id = ThreadsafeCountersFile::next('transaction_id');

Increment by a value other than 1...

$unique_transaction_id = ThreadsafeCountersFile::next('transaction_id', 10);

Because the file implementation makes use of files on disk to track the last value we need to be able to handle the case of a file deletion. Deleting the counter file will reset the counter but if our application is tracking its last use of the generated value (perhaps by storing it somewhere else) then we can have our next() routine check that the next value it generates is greater than the last noted value. If it isn't then the file has probably been deleted and we will need to generate an id that is ahead of the last possible value.

$last_used_value       = getLastUsedId();
$unique_transaction_id = ThreadsafeCountersFile::next('transaction_id', 1, $last_used_value);

Using The Redis-Based Implementation

Basic use of Redis implementation...

require_once "ThreadsafeCountersBase.php";
require_once "ThreadsafeCountersRedis.php";
$unique_transaction_id = ThreadsafeCountersRedis::next('transaction_id');

Increment by values greater than 1 is also possible...

$unique_transaction_id = ThreadsafeCountersRedis::next('transaction_id', 1000);

As with files, if the redis key being used to store the counter is deleted or changed to a value that has already been issued we can catch the problem and recover from it if we are keeping a track of the values already issued.

$last_used_value       = getLastUsedId();
$unique_transaction_id = ThreadsafeCountersRedis::next('transaction_id', 1, $last_used_value);

It is up to your application to track the last used Id in some way (perhaps it is being used as a key in a DB column!)

ProcessWire

There are module files provided for users of the ProcessWire CMF/CMS platform. One each for the files and Redis implementations.

Files

The files implementation creates a new directory for your counters; "assets/counters" and stores all its counter files in there. As long as your assets/ directory is writable this should all work straight away. You can check the module settings when it is installed as it will tell you about any corrective actions needed.

Redis

The Redis module allows you to configure the IPv4 address and port of the redis server you wish to store your counters on. Set these up as required.

Usage

To use the counters from your template files you need to load the counter module and then call the next() method.

$countserver = $modules->get('ThreadsafeRedisCounters'); // Use 'ThreadsafeFileCounters' to use files instead.
$next_id     = $countserver->next('your_counter_name');

Copyright

Copyright (c) 2014 Netcarver

The above copyright notice and the following permissions and disclaimer shall be included in all copies or substantial portions of the Software.

DO NOT DISTRIBUTE OTHER THAN ACCORDING TO THE TERMS OF YOUR LICENSE FILE

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

escrow.md

The Great Shopping Race

Have you ever been in a shop during the new-year sales? There can be a lot of demand for a limited number of physical products but there can never be more items on the shelves and in shopping baskets than the actual physical number of items there were for sale in the first place as the atoms of the items themselves prevent duplication.

It's not quite the same for virtual shops; there are no physical products in the webserver - just bits representing them. By careful programming we have to provide safeguards against race conditions ourselves. Threadsafe counters provides one method for doing this based on an escrow model in which sales are not a 2-state affair (available or sold) but each counter is actually a 3-tuple with a primary value, a reserved (escrowed) value and a confirmed (sold) value. You don't have to worry about these values - they are hidden behind an easy-to-use API that takes its names from the physical sales arena.

Let's make this a little more concrete and imagine we want to sell a limited supply of tickets to a ProcessWire meetup. Ryan's booked a venue with 100 seats so we'll be selling a maximum of 100 tickets. Trouble is, these things sell quickly and we don't want to over-sell the seats so we need a system that guarantees we can't do that. And TS counters can provide that guarantee.

Initialisation

We need to create a TS Counter object to access our counts.

// Setup a counter using the filesystem or Redis if you prefer...
$pw_tickets = new ThreadsafeFileCounter('PW-NYC-2014-10');

// Setup a counter using Redis...
//$pw_tickets = new ThreadsafeRedisCounter('PW-NYC-2014-10');

And, at some point, we'll need to set the initial "stock" level of our product. This could be a physical product or any other resource that has a countable, finite, supply. In our case, we'll initialise our ticket counter to have 100 available tickets.

// We initially have 100 tickets available...
$pw_tickets->init(100, true);

So, how many are available?

You can read and display the instantanious availability level for display in product listings using the following;

$available = $pw_tickets->available();

This is going to be useful on our page about the meeting.

Should we ever need to show it, we can also access how many tickets are currently reserved pending payment completion;

$reserved  = $pw_tickets->reserved();

And if you want to show how many sales have successfully completed, that's easy too;

$total_sold = $pw_tickets->completed();

NB tickets aren't considered "sold out" until the available count and the reserved count both hit zero and you can test this directly with the exhausted() method. Like this;

if ($pw_tickets->exhausted()) {
    ...
}

The difference between having the $available count hit zero and the supply totally exhausted is that if there are tickets in the purchase process, a failed payment can release some reserved tickets back into the available pool (more on this in a while.)

Getting Items Into Escrow

When shop visitors try to purchase tickets they are, in effect, putting their wanted number of items into escrow. They haven't paid for the tickets just yet but the tickets are now "spoken for", or reserved for a while, and can no longer be sold to other potential buyers until this customers payment either succeeds or fails.

Its up to your application to decide when items should be put into escrow using the reserve() method. You might want to reserve things as they are added to a shopping basket or you might want to reserve them when the checkout process starts. Either way, you always need to check the return value from the reservation call as it tells you how many of the requested item were successfully reserved and this can be different to what you actually requested!

// try to reserve 5 tickets - but we may get 0 if all 5 are not available to us...
$my_ticket_count = $pw_tickets->reserve(5);
if ($my_ticket_count) {
    // We got all 5 we asked for so we can add that number to our basket ready for payment.
    ...
} else {
    // Could not reserve all 5 tickets. Perhaps someone got them just before us, or perhaps there are only 4
    // so our reservation can't be fully honoured.
    ....
}

In some applications or cases, people might be OK with getting less than the total number they'd prefer to reserve. In that case the API allows partial reservations as follows...

$my_ticket_count = $pw_tickets->reserve(5, ThreadsafeCountersBase::ALLOW_PARTIAL_XFER);

In the last case we can have anything from 0 to 5 in $my_ticket_count and your application logic will need to handle all cases.

Completing A Sale

Some time after putting $my_ticket_count quantity of tickets into escrow the potential buyer will either successfully pay for the order or payment will fail for some reason. If the payment succeeds you can complete the order process by moving your ticket count out of escrow to the completed state using the complete() method.

$pw_tickets->complete($my_ticket_count);

Once complete, the given number of reserved tickets can never be returned to the reserved or available counts as 'completed' represents the total number of that product ever sold.

Failed Payments

If a payment does fail, you just release your reserved count back from escrow to the available ticket pool...

$pw_tickets->release($my_ticket_count);

Managing Stock Levels

Whenever a new supply of your product comes in (perhaps Ryan books a larger venue in the case of PW meetup tickets) you will want to boost the available number of tickets by the additional number available.

// Ryan books a hall with seating for 200 people, an increase of 100 seats...
$pw_tickets->restock(100);

In the case where you have to pull stock from sale or reduce availability for some reason, you can use withdraw() but remember to check the return value to see if your managed to reduce the amount as much as you wanted. If the available count is lower than the withdraw request then you might need to take action.

// The 100-seat hall cancels and Ryan has to book a hall with 80 seats instead; a 20 seat reduction...
$reduction = 20;
$withdrawn = $pw_tickets->withdraw($reduction);
if ($withdrawn < $reduction) {
    // Could not reduce by 20 seats... take action!
}

In our case, perhaps Ryan had already reached the point where there were 90 tickets either sold or reserved, leaving only 10 available. The withdrawing of 20 seats is no longer possible and the code would need to signal the need to book extra seats or somehow accommodate the additional, already sold places.

Returns

Handling customer returns is not supported directly in the API because, in most cases, the returns process cannot be automatic. Some form of manual intervention is often needed for physical goods at least. However, you can model returns by restocking by the number of items returned once the returned items have been inspected and found good enough to re-sell.