I came up with what I think is kind of an interesting solution to a problem we were having on a work project. Problem was that there's a big chain of calculations that need to be done, with different services for different steps. Each one reads from and writes to several database tables. We wanted the web services to automatically invoke all the appropriate steps to recalculate data down the chain when data up the chain was modified.

To make things a little more complicated, some steps are asynchronous. They are implemented by a Python worker on some other server reading jobs out of some Amazon queue. So we can't simply have a series of functions that we call in order. There needs to be a way for these asynchronous jobs to call one of our services to say 'hey, I finished step 3', and have us complete the chain.

The initial approach that we started using (which I was never entirely sold on) was to have each step invoke the next when it was done doing its work. But this results in not being able to test each step independently. Try to test step 1 and you end up running steps 2, 3, and 4, also (which is especially troublesome when you're trying to run steps 1-3 in order to set up for testing step 4). Another trouble with this approach is that the logic for determining what to do next is scattered throughout all the different services (including some in another codebase!).

Another approach would be to keep 'last update' timestamps on everything and be careful to always update them, and then just run a 'recalculate all the outdated data' task once in a while. I don't like this approach because the database schema is already annoyingly complex and interconnected (enough so that it's difficult to keep the entire thing in your head in order to reason about it), and keeping those timestamps on everything would just add to the complexity without making the process any more obvious.

In an ideal world I suppose there would be some kind of formal dependency managment system that we would use to make sure that down-chain data is updated when it needs to be (preferrably atomically and without race conditions!), but this project's architecture ship (or rather its lack-of-architecture ship) has pretty much sailed already, so I needed to come up with a less invasive and quicker-to-implement solution.

So the interesting solution that I've come up with is this: Each service in the chain is still in charge of determining what needs to be done next (though in an attempt to make the high-level flow more obvious I've created a central 'workflowator' object to which they delegate to help make this determination). They just don't actually execute those next steps themselves. Instead, they enqueue a list of actions to be invoked later on the action context (an object that's passed into all service implementation functions). The unit tests that invoke the service functions directly can ignore these queued jobs, but the normal request handling process will keep running them until the queue is empty. Because of the way Phrebar services are implemented — each service basically being a pair of functions that construct and invoke an 'action' (in practice this is implemented as a RESTAction class with a constructor and an __invoke($actionContext) method) — this worked out really easily; queued jobs implement the same interface as our services, so we can re-use the same RESTAction classes to implement them.

i.e. implementations of services end up following this pattern:

   class BaseAction implements TOGoS_Action { ... }
   
   class Action1 extends BaseAction
   {
      // We'll be reading and writing various records related to the one identified by:
      protected $idOfObjectBeingRecalculated;
      
      public function __construct( $registry, $idOfObjectBeingRecalculated ) {
         parent::__construct($registry);
         $this->idOfObjectBeingRecalculated = $idOfObjectBeingRecalculated;
      }
      
      public function __invoke( $actionContext ) {
         ...do some work that updates the database...
         
         // Some other precalculated aspect of our object
         // is now invalid, but action2 will fix it,
         // so queue that up.
	 $actionContext->queueJobs( [
            new Action2( $registry, $this->idOfObjectBeingRecalculated )
         ] );
         
         // Return an HTTP 204 to indicate that this step succeeded
         return $this->jsonResponse( 204 );
      }
   }