IllicoDB3 Evolutions
A lot has transpired since the ten articles from Quitus, particularly regarding the Small Business Suite—a business suite that is still a work in progress although the fact that we successfully fulfilled the initial contract to develop four foundational modules, all four modules completed in just six days. We even extended our efforts by an additional four days to demonstrate how each module could be made available as web components, complete with bidirectional communication between components, reminiscent of Angular's functionality.
It's only natural that IllicoDB3 has evolved alongside the Quitus suite. This is a common occurrence: when your product reaches any stage of production, it naturally adapts to the genuine needs expressed by end-users—even if you are one of those users! Trust me when I say this: it’s ALWAYS like this. This is precisely why achieving a production-ready state as quickly as possible is desirable; testing your product under real-world conditions accelerates its evolution and allows you to prioritize tasks based on user feedback. I encourage you to establish direct feedback loops between end-users and your development team, including operations teams (Ops, hence DevOps or DevSecOps) who are also users with their own unique constraints. Listening closely to them can significantly ease their workload. These are fundamental principles of agility that I find essential to reiterate here, especially as we work on a tangible product—even if it's just an example.
when your product reaches any stage of production, it naturally adapts to the genuine needs expressed by end-users. It’s ALWAYS like this!
Speaking of which, let me present you with a complex diagram. It may appear intricate, but it’s not complicated. I've highlighted what I believe to be crucial elements. The two sections circled in red represent the information flows coming from your end-users; notably, there are yellow waves radiating from the clients (the small group of blue figures below in the diagram). These waves indicate at least two things: (1) you do not control what your clients say about you or your products and (2) you can channel part of this information flow directly to your development team (as shown by the orange/yellow arrows leading to the team, including its Product Owner and the team's backlog—an expression I prefer over "Product Backlog," especially when a single team works on multiple products, which is almost always the case when assembled for a long time). This also applies to iteration backlogs if you're working in timeboxes (note that it’s quite similar when using Kanban).
These arrows extend even further (the section circled in blue), often through Customer Service or support, and can reach higher levels within the organization.
I will gloss over the section circled in red on the right side of the diagram, which is essential when developing beta phases. We will likely revisit this later.
What matters in this diagram is understanding that intercepting messages from end-users as directly as possible is in your best interest. This aligns with pillar #5 of Digital Transformations, which is illustrated in the second diagram I present below. One of the key pillars of Digital Transformations is to embrace cycles that become increasingly direct—and hopefully shorter—leading to enhanced satisfaction for clients/users and facilitating your company's sales and marketing processes.
What Evolutions
The needs that emerged from the development of Quitus have been chaotic and multifaceted for IllicoDB3, resembling a vortex. This is normal: the requirements expressed by users with diverse interests are like a big spoon stirring the soup. It is futile to try to organize them. However, prioritizing these needs is not in vain, although it is far from easy, especially if the Product Owner (PO) only embraces a functional view. Having a PO who understands both the business and technical aspects is a significant advantage, making them, in my opinion, a rare find. Ignoring the purely functional side is an error, just as it is a mistake to focus solely on the technical side. Additionally, the PO must possess the experience required to determine when refactoring is necessary, which further sharpens their responsibilities. I will also add that they must assess the priority of resolving a bug in relation to the entire backlog (You Build It; You Run It). No, being a PO is not an easy job—not by a long shot.
So, what new features have been revealed by Quitus? I will list them for you in the order they appeared.
An ID should be automatically generated when adding a new record. An ID should be read-only once established. Therefore, we need read-only fields!
Since an IllicoDB3 table consists of records with a fixed length, modifying a structure becomes challenging. With Quitus, we faced our share of modifications, particularly with the products table, which has evolved multiple times. To avoid losing the data we had in the database, we had to make multiple copies between the old table and the new one. This slows down development! We need to be able to alter a table's structure much more easily. However, creating the necessary code was deemed important; thus, I preferred to invent a "reserved" field type that would allow us to freeze part of the content in a record at a configurable length. It also seemed wise not to allow modifications to this type of field: it should be able to remain empty, which also makes it a read-only field.
Next, still starting from the product table, it became clear that it was important to be able to link a table with satellite tables (basic feature of a DB engine), such as linking the product table with the stock table. Therefore, we created a field of type 'fk' (Foreign Key).
The existing 'gender' field seemed like a good candidate to be an 'enum' type field with three possible values: 'F', 'M', or '?'. So, we created a field of type 'enum'.
But what should be the default value for a 'gender' field? Well... '?' seems appropriate! Thus, we will allow specifying a default value for certain fields. And there you have a new feature.
Next, if fields have default values, we need to ensure that these default values are actually encoded in the record when it is created: modifying the 'append' method became a necessity.
When displaying a browse of invoices, it’s better to present the invoices from the most recent to the oldest. Oh really... yes, I should have guessed that! But don’t I have an 'index' method? Yes, that's true, but it always operates in ascending mode. However, here I need it to be in descending mode. Boom... we need to modify the 'index' method.
Since we have plenty of test code, we realize that searches (the 'seek' method) no longer work. Well, yes, moving forward or backward in the search must be based on the mode: ascending or descending. And boom, here we are modifying the search function ('seek').
Once again, the famous 'products' table! I mentioned that I base my work on what schema.org provides: Product. And wow... we have some quite specific attributes, such as gtin, gtin12, gtin13, mpn, nsn, etc.
Here is a list of new types of fields:
$types['country' ] = 2; /* ISO 3166-2 */
$types['currency' ] = 3; /* ISO 4217 */
$types['duns' ] = 9; /* This term uses terminology from the GoodRelations Vocabulary for E-Commerce */
$types['email' ] = 64; /* auto verified */
$types['enum' ] = 64; /* Enum
$types['fk' ] = 32; /* Foreign key */
$types['gender' ] = 1; /* F, M or ? ... is an enum with imposed values */
$types['gln' ] = 13; /* https://www.gs1.org/gln */
$types['guid' ] = 48; /* sha-384 length */
$types['isic' ] = 10; /* International Standard Industrial Classification of All Economic Activities (ISIC) (https://ilostat.ilo.org/methods/concepts-and-definitions/classification-economic-activities/) */
$types['lang' ] = 5; /* Language (eg. US-en) */
$types['lupdate' ] = 14; /* Like datetime but read-only */
$types['password' ] = 50; /* Something tha can not be shown and whose value is a hash */
$types['phone' ] = 30; /* Phone number - seemls to be enough for all countres */
$types['uint' ] = 4; /* Not used at the moment */
$types['url' ] = 255; /* URL - not long enough but that should make it */
$types['vat' ] = 14; /* 14 bytes is enough for all countries in the world; shoulmd be auto-verified */
If we want to maintain a minimum level of control over the records in a table, whether through an audit trail view or a business intelligence perspective, it becomes essential to add technical fields to each table, preferably using the exact same fields from table to table. This necessity leads to the establishment of templates (sets of fields) that need to be repeatedly inserted into each table within a database. This is how the concept of 'template' emerged in the structure of a database.
It seemed wise to have a consistent description for a field, especially when it appears in multiple tables. And thus, the concept of definition was born (which we will discuss later).
领英推荐
With a large number of tables in the same database, it becomes essential to document the database. This is something that cannot be overlooked in a real project. Thus, we invented the "document" method for a database, a method that generates very useful HTML.
With Quitus, we aimed to present a module in a simple way: a browse of the records from the main table of the module, and below the browse, a form that allows for modifying a record (this kind of interface is inherited from the ?Je Gère.../I Manage...? software collection my previous company wrote a while ago). It was at this point that the need to synchronize the browse with the form became apparent (when switching from one record to another, the data in the form must also be updated). For Quitus, we therefore created a fairly standardized "browse" method and a "__toForm" method that generates a form with all the necessary fields. And thus, two new methods "browse" and "__toForm" were born.
Recent needs have made additional methods necessary. The latest one involves creating temporary tables where fields from one table can be copied into the temporary table. However, since some fields are pre-filled (for example, the "id" field), we need to be able to override the "read-only" rule to ensure that the value in the "clone" table matches exactly with that in the master table.
As you can see, Quitus has significantly evolved IllicoDB3. We won't be able to cover ALL the new features all at once. Therefore, we will do this over the course of 3 or 4 successive articles.
Cela m'amène à parler un peu des estimations et du travail qui est prévu (celui qui se trouve dans le backlog). Je pense que vous allez être surpris car en effet s'il est habituel de prévoir 3, on peut se dire qu'un tiers nous a échappé (1/3) et que ce tiers provoque encore des impacts en cascade pour la moitié ! On doit donc penser en ces termes:
What We Estimate Compared to What We Overlook
This brings me to discuss a bit about estimates and the work that is planned (the tasks in the backlog). I think you will be surprised because, while it is common to estimate 3, we might realize that one-third has slipped through our fingers (1/3), and that this third still causes cascading impacts for the other half! Therefore, we need to think in these terms:
So we might think that instead of estimating 3, we should estimate 4?. But no... it’s not that simple! We need to view this in two dimensions, as follows:
Let's take 3 circles that respect these proportions: 60 cm radius at the center, 80 cm radius in the middle (which is indeed 60 + 60/3), and 90 cm on the outside.
Looking at the diagram, you might think that you have covered the majority. However, think again, because applying the formula for the area of a circle, C1 has an area of 11309.73 cm2, C2 has an area of 20106.19 cm2, and C3 has an area of 25447.05 cm2. Therefore, the area that is overlooked (C3 - C1) represents 14137.32 cm2, which is much larger than 11309.73 cm2 (C1). We can conclude that we overlook more than we estimate! This is quite counterintuitive. This once again advocates for an early friction between what has been built (and envisaged) and the reality of production (the real world) so that the most significant shortcomings are revealed as quickly as possible.
Let's start with the ones that seem to me the most influential...
Templates and Definitions
We had to modify the way a database is created, how its structure is read, and adapt the IllicoDB3, IllicoDB3Table, and even IllicoDB3Field classes. I will just show you what it looks like for the creation of the database and the reading of such structure:
/* ==================================================== */
/** {{*generateField( &$fieldNames,&$field )=
Generates a field definition based on the provided field names and field properties.
{*params
&$fieldNames (array) An array of field names that will be used to define
the structure of the field.
&$field (array) An associative array containing the properties of
the field (e.g., type, length, default value).
*}
{*return
(string) The xml code that corresponds to the creation of the field
*}
{*warning
This method is private!
*}
{*mdate 14/09/2024 10:07:55 *}
{*todo
1) Implement validation for field properties to ensure they meet
database standards.
2) Allow for additional field attributes (e.g., unique, indexed) to
be specified.
*}}
*/
/* ==================================================== */
private static function generateField( &$fieldNames,&$field,&$definitions )
{
if ( isset( $fieldNames[ $field['name'] ?? '====='] ) )
throw new \Exception( __METHOD__ . "() at line " . __LINE__ . ": duplicate field ({$field['name']}) (ErrCode: ILLICODB3_EXCEPTION_DUPLICATE_FIELD = " . ILLICODB3_EXCEPTION_DUPLICATE_FIELD . ")",ILLICODB3_EXCEPTION_DUPLICATE_FIELD );
else
$fieldNames[ $field['name'] ] = true;
if ( ( $iLength = self::fieldLength( $field['type'] ) ) === -1 )
if ( isset( $field['length'] ) )
$iLength = $field['length'];
else
throw new \Exception( __METHOD__ . "() at line " . __LINE__ . ": invalid field length for {$field['name']} (ErrCode: ILLICODB3_EXCEPTION_INVALID_FIELD_LENGTH = " . ILLICODB3_EXCEPTION_INVALID_FIELD_LENGTH . ")",ILLICODB3_EXCEPTION_INVALID_FIELD_LENGTH );
elseif ( $iLength === -ILLICODB3_EXCEPTION_INVALID_FIELD )
throw new \Exception( __METHOD__ . "() at line " . __LINE__ . ": invalid field type ({$field['type']}) (ErrCode: ILLICODB3_EXCEPTION_INVALID_FIELD = " . ILLICODB3_EXCEPTION_INVALID_FIELD . ")",ILLICODB3_EXCEPTION_INVALID_FIELD );
if ( ! isset( $field['label'] ) )
$field['label'] = '';
if ( ! isset( $field['comment'] ) )
$field['comment'] = '';
else
$field['comment'] = trim( $field['comment'] );
if ( ! isset( $field['default'] ) )
$default = null;
else
$default = "default=\"{$field['default']}\"";
if ( ! isset( $field['values'] ) )
$values = null;
else
{
$x = explode( ',',str_replace( ';',',',$field['values'] ) );
$values = '';
foreach( $x as $y )
$values .= trim( $y ) . ',';
$values = "values=\"" . v::STR_strin( $values ) . "\"";
}
// Sometimes we have the same kind of field iin multiple tables
// Instead of repeating the same comment over and over again
// we can simply refer to a definition
if ( preg_match( '/\%definition:(?P<name>.+)\%$/s',$field['comment'],$matches ) )
{
if ( ! empty( $definitions ) && isset( $definitions[ $matches['name'] ] ) )
$field['comment'] = $definitions[ $matches['name'] ];
}
return ( SP_12 . "<field " . str_pad( "name=\"{$field['name']}\"",40 ) . " length=\"{$iLength}\" type=\"{$field['type']}\" {$default} {$values} label=\"{$field['label']}\" comment=\"" . htmlentities( $field['comment'] ) . "\" />\n" );
} /* End of IllicoDB3.generateField() ======================= */
/* ==================================================== */
/** {{*create( $DBName,$folder,$tables )=
Create a DB in the @param.folder directory with the @param.tables tables
{*params
$DBName (string) The name of the DB (which is also going to be
the name of the physical file in which the DB
structure will be saved)
$folder (string) The folder in which the DB structure will be
written in
$tables (array) An array of tables that must be described in
the DB structure
*}
{*return
(boolean) [c]true[/c] if the method is successful; [c]false[/c] otherwise
*}
{*todo
1) Have the possibility to give a description to the DB
*}
*}}
*/
/* ==================================================== */
static public function create( string $DBName,string $folder,array $tables,array $templates = null,array $definitions = null )
{
$DBFile = "{$folder}/{$DBName}" . ILLICODB3_EXTENSION;
$szXML = '<?xml version="1.0" encoding="UTF-8"?>' . "\n";
$szXML .= "<database name=\"{$DBName}\">\n";
$szXML .= SP_4 . "<tables>\n";
/* If we have defined some terms, here they are: we load them in an array of definitions */
if ( ! empty( $definitions ) )
{
foreach ( $definitions as $key => $value )
{
$szXML .= SP_8 . "<definition name=\"{$key}\">{$value}</definition>\n";
} /* foreach ( $aTables as $name => $aData ) */
$szXML .= SP_8 . "<!-- ****************** -->\n";
} /* if ( $definitions ) */
/* ------------------------------------------------------------------------------------ */
/* If we have defined templates (sets of predefined fields), here they are: we load
them in an array of templates */
if ( ! empty( $templates ) )
{
foreach ( $templates as $key => $template )
{
$szXML .= SP_8 . "<template name=\"{$key}\">\n";
$fieldNames = [];
foreach( $template['fields'] as $field )
$szXML .= self::generateField( $fieldNames,$field,$definitions );
$szXML .= SP_8 . "</template>\n";
$szXML .= SP_8 . "<!-- ****************** -->\n";
} /* foreach ( $templates as $key => $template ) */
}
/* ------------------------------------------------------------------------------------ */
/* If we have defined tables here they are: we load them and generate the XML
corresponding to each table */
foreach ( $tables as $key => $tableData )
{
$szXML .= SP_8 . "<table name=\"{$tableData['name']}\">\n";
$fieldNames = [];
foreach( $tableData['fields'] as $field )
{
if ( isset( $field['template'] ) )
{
$templateName = $field['template'];
//var_dump( $templates );
if ( isset( $templates[ $templateName ] ) )
{
$template = $templates[ $templateName ];
foreach( $template['fields'] as $field )
$szXML .= self::generateField( $fieldNames,$field,$definitions );
}
continue;
} /* if ( isset( $field['template'] ) ) */
$szXML .= self::generateField( $fieldNames,$field,$definitions );
} /* foreach( $aData['fields'] as $aField ) */
$szXML .= SP_8 . "</table>\n";
$szXML .= SP_8 . "<!-- ****************** -->\n";
} /* foreach ( $tables as $key => $tableData ) */
$szXML .= SP_4 . "</tables>\n";
$szXML .= "</database>\n";
//return ( v::FIL_StrToFile( $szXML,$DBFile ) );
return ( self::strToFile( $szXML,$DBFile ) );
} /* End of IllicoDB3.create() */
/* ==================================================== */
/** {{*readStructure()=
Reads the internal structure of the DB
{*params
*}
{*warning
Used internally by the class (method is protected)
*}
{*return
(self) Returns the current instance of the class
*}
*}}
*/
/* ==================================================== */
protected function readStructure()
{
if ( $this->isOpen )
{
if ( is_file( $DBFile = self::addTrailingSlash( $this->homeOfData ) . $this->name . ILLICODB3_EXTENSION ) )
{
$oDom = new \DOMDocument();
if ( $oDom->load( $DBFile ) )
{
if ( ( $oXPath = new \DOMXPath( $oDom ) ) && method_exists( $oXPath,'query' ) )
{
if ( ( $tableCollection = $oXPath->query( 'tables/table' ) ) && ( $tableCollection->length > 0 ) )
{
$this->tables = [];
$tableModel = new IllicoDB3Table();
$tableModel->container = clone $this;
$fieldModel = new IllicoDB3Field();
foreach( $tableCollection as $table )
{
$thisTable = clone $tableModel;
$thisTable->name = trim( $table->getAttribute( 'name' ) );
if ( ( $fieldCollection = $oXPath->query( 'field',$table ) ) && ( $fieldCollection->length > 0 ) )
{
$thisTable->fields = [];
$offset = 4;
foreach( $fieldCollection as $field )
{
$thisField = clone $fieldModel;
$thisField->name = $field->getAttribute( 'name' );
$thisField->length = (int) $field->getAttribute( 'length' );
$thisField->type = $field->getAttribute( 'type' );
$thisField->label = $field->getAttribute( 'label' );
$thisField->comment = $field->getAttribute( 'comment' );
$thisField->default = $field->getAttribute( 'default' );
$thisField->offset = $offset;
$offset += $thisField->length;
if ( $thisField->type === 'enum' )
{
$thisField->values = explode( ',',$field->getAttribute( 'values' ) );
$thisField->values = array_combine( $thisField->values,$thisField->values );
//var_dump( $thisField->values );
}
$thisTable->fields[$thisField->name] = $thisField;
if ( $thisField->type === 'id' )
$thisTable->hasID = $thisField;
} /* foreach( $fieldCollection as $field ) */
$this->tables[$thisTable->name] = $thisTable;
}
} /* foreach( $tableCollection as $table ) */
} /* if ( ( $tableCollection = $oXPath->query( 'tables/table' ) ) && ( $tableCollection->length > 0 ) ) */
} /* if ( ( $oXPath = new \DOMXPath( $oDom ) ) && method_exists( $oXPath,'query' ) ) */
else
{
throw new \Exception( __METHOD__ . "() at line " . __LINE__ . ": 'INVALID XPATH OBJECT (query() method NOT FOUND) (ErrCode: ILLICODB3_EXCEPTION_INVALID_DB_CONTAINER = " . ILLICODB3_EXCEPTION_INVALID_DB_CONTAINER . ")",ILLICODB3_EXCEPTION_INVALID_DB_CONTAINER );
}
}
else
{
throw new \Exception( __METHOD__ . "() at line " . __LINE__ . ": 'INVALID DB CONTAINER (cannot be loaded) (ErrCode: ILLICODB3_EXCEPTION_INVALID_DB_CONTAINER = " . ILLICODB3_EXCEPTION_INVALID_DB_CONTAINER . ")",ILLICODB3_EXCEPTION_INVALID_DB_CONTAINER );
}
}
else
{
throw new \Exception( __METHOD__ . "() at line " . __LINE__ . ": '{$DBFile}' DB NOT FOUND (homeOfData = " . self::addTrailingSlash( $this->homeOfData ) . "; name = '{$this->name}') (ErrCode: ILLICODB3_EXCEPTION_DB_CONTAINER_NOT_FOUND = " . ILLICODB3_EXCEPTION_DB_CONTAINER_NOT_FOUND . ")",ILLICODB3_EXCEPTION_DB_CONTAINER_NOT_FOUND );
}
} /* if ( $this->isOpen ) */
else
{
throw new \Exception( __METHOD__ . "() at line " . __LINE__ . ": database NOT open (ErrCode: ILLICODB3_EXCEPTION_DB_NOT_OPEN = " . ILLICODB3_EXCEPTION_DB_NOT_OPEN . ")",ILLICODB3_EXCEPTION_DB_NOT_OPEN );
}
return ( $this );
} /* End of IllicoDB3.readStructure() */
It goes without saying that new exceptions have been defined.
Conclusion
This will be all for today. We have seen many new things, even though we have only barely scratched the surface of the code that has been modified. We have explored quite a few concepts that lend weight to the arguments for Agility, and that is very good. IllicoDB3 is equipped with all new features that will be put to use very quickly. These features come from a friction with the reality on the ground, a reality that is the only one capable of guiding us on what needs to be done versus what does not need to be done, or at least not right now.