AutoPatch README

--------------------------------------------------------------------------

Table of Contents

0) Where do I get AutoPatch?
1) What is AutoPatch?
2) Design Requirements
3) Basic Design
4) Detailed Design
5) How to integrate AutoPatch with a web application
6) How to integrate AutoPatch with your development cycle
7) How to implement a patch
8) How to extend AutoPatch with new patch types
9) How to rollback patches
10) How to migrate from a previous version of AutoPatch to 1.2

--------------------------------------------------------------------------


0) Where do I get AutoPatch?
----------------------------

AutoPatch is open source, and is developed transparently on SourceForge.

You may find more information at http://autopatch.sourceforge.net

There is a tracker for bugs, feature requests, and patches.
There are mailing lists for announcements, users, and cvs commits.

You can browse the CVS repository and watch the development if you want.

--------------------------------------------------------------------------


1) What is AutoPatch?
---------------------

AutoPatch is an automated Java patching system. Among other things, it
allows a Java application to easily do things like keep a database
schema in sync with what new application versions require, thus
radically lowering the cost of database maintenance.

Thought of abstractly, you can think of a "patch" as any change that
must be applied to any system external to your Java application.
AutoPatch is a system that automates the application of these patches.

That allows you to use rapid methodologies (e.g. XP, spiral development
model) all the way down to the alteration of external systems, without
needing an army of DBAs or similar to go around and patch your databases 
when you release new code requiring schema changes.

Owing to a semi-recent name change, you will find uses of the word
"migrate" or "migration" everywhere. Please consider the words
"migrate" and "migration" as synonyms for "patch".

--------------------------------------------------------------------------


2) Design Requirements
----------------------

AutoPatch was designed with a few very simple requirements in mind.

o  Patches should execute before any server logic
     - This is important, or the application logic couldn't trust the
       state of the external systems
o  AutoPatch must enforce a total natural ordering on all patches.
     - This allows a developer to depend on the results of previous
       patches, for instance a patch can use database tables defined 
       in patches that were executed previously
o  AutoPatch must ensure that each patch is applied once and only once
     - Having them apply multiple times would mean the patch author
       could never trust the initial state of the database, making
       patch development very difficult
o  AutoPatch should be able to globally order patches that execute
       against multiple external systems
o  AutoPatch should be capable of executing post-patching maintenance
       after patching (things like recompiling pl/sql, etc)
o  AutoPatch should be capable of executing the same patch across
       multiple databases with the same schema
o  AutoPatch should be capable of running in a read only way, showing
       what it would do, but not changing anything
   

--------------------------------------------------------------------------


3) Basic Design
---------------

Given the AutoPatch design requirements, a very simple design has been
created. There is a patch launcher, which serves as the controller
for all patch activity.

3a) Startup
-----------
Applications wishing to use AutoPatch must instantiate a
MigrationLauncher by implementing an adapter that starts the launcher
on application startup, or using an existing adapter.

3b) Patch Level
---------------
The launcher is able to determine the current patch level of the system
in a durable way, so that it knows the patch level across multiple runs.

3c) Patch Loading
-----------------
The launcher searches the classpath for any available objects that
implement a common patch type, or any files that are in a
directory known to hold patches.

3d) Patch Ordering
------------------
The launcher loads all of the patches into a single list and orders them.

3e) Patch Execution
-------------------
Each patch is executed, one after another. Any errors are considered
fatal, as the application logic will not be able to trust the state of
external systems, and is thus cause for total shutdown.

3f) Post-Patch Tasks
--------------------
These have loading, ordering and execution steps similar to regular patches,
the only difference is that these run *every* time AutoPatch runs. This allows
you to do maintenance tasks (like recompiling PL/SQL or similar) after patch
runs, but means that the tasks in here *must* be re-runnable. They will run a lot.

--------------------------------------------------------------------------


4) Detailed Design
------------------

Expanding on the Basic Design with implementation-specific details,
this section describes the current implementation of AutoPatch, with
references to actual classnames.

4a) Startup
-----------
The MigrationLauncher object is the main entry point of the AutoPatch
system. There are a few adaptors that have been implemented, one for web
application integration (WebAppMigrationLauncher), one for
command-line integration (StandaloneMigrationLauncher), as well as
a few for depencency injection containers (AutoPatchService). There is also
a "distributed" version of each launcher that allows you to configure
multiple sub-launchers together as one large system orchestrated as a single
unit with a single patch level. 

The sole purpose of the adaptors is to instantiate the MigrationLauncher object,
and call the "doMigrations" method.

Once the StandaloneMigrationLauncher is executing, it will read a configuration
file from the classpath called migration.properties. An example
migration.properties is provided in the same directory as this
README. The configuration file is responsible for configuring the
database connection for the AutoPatch system, as well as for
specifying where in the classpath AutoPatch should search for patches.

Other MigrationLaunchers are configured in different ways, and you should
consult the javadoc information in the adapter that you wish to use.

4b) Patch Level
---------------
Once the MigrationLauncher has started executing, and finished reading
in its configuration file, it will use the database parameters to
configure the PatchTable object. PatchTable will inspect the database
and attempt to get the current patch level of the system. If no patch
table is present, it will create one (named 'patches').

Currently, the configuration file specifies a "database type", and
the name of the type specifies which SQL property file to load, so
PatchTable can interact with multiple database types. For example, if
the type is "postgres", PatchTable will attempt to load the SQL
property file "postgres.properties" and use the SQL in that file to
interact with the database when persisting patch level information.

4c) Patch Loading
-----------------
Once the patch level is determined, the colon-separated patch path
specified in the configuration file is searched for all objects that
implement the MigrationTask type. Each directory in your classpath has
each part of the patch-path appended to it, and each of these combinations
is searched for patches.

There are a couple of MigrationTask sub-classes that are useful to extend, 
and in practice all patches use the sub-classes.

The first sub-class is the SqlScriptMigrationTask. It allows for raw
text files containing SQL to be used as migrations, so long as they
follow the naming convention 'patchNNNN_<name>.sql'
(e.g. 'patch0001_create_initial_schema.sql'), where the numbers are
globally unique among all patches.

The second sub-class is the SqlLoadMigrationTask, which allows you to
quickly implement domain data loads with the use of tab-delimited text
files and a sub-class that understands how to insert one row of the
data at a time.

Another data load type is the FlatXmlDataSetTaskSource.  This class allows
you to use xml data files to bulk load data into the system.  As per the
name, the xml format is the same as the DBUnit FlatXmlDataSet format
type (http://dbunit.sourceforge.net/apidocs/org/dbunit/dataset/xml/FlatXmlDataSet.html).  
Data sets in this format must match the typical patch naming
standard, except that it must end with a .xml suffix (patchNNNN_<name>.xml).
Examples:
patch0002_initial_data.xml
patch0005_load_stuff.xml

Another sub-class currently implemented is
MigrationTaskSupport. This is a more basic patch object, and allows
for any arbitrary logic to be implemented in Java and executed under 
the guise of a patch. It is potentially very useful to perform a data 
migration after one patch adds new tables and columns, but before another 
patch removes other tables or columns.

4d) Patch Ordering
------------------
Its important to realize that patches must have a total global ordering -
meaning that each patch must be associated with a unique level.

Java patches get the opportunity to specify their own level, but for
the sql and xml patches, ordering is based on the naming convention
specified above. This may seem cavalier at first, but its simple and easy
to do in practice so long as you check the current patch level before
creating a new patch, and coordinate with your fellow patch authors if
there are multiple developers working on a system.

To determine the current patch level easily, its possible to execute 
the MigrationInformation object as a standalone Java application, 
which will tell you the current patch level in the database, and the 
current maximum patch level implemented by MigrationTasks in the classpath. 
A suggested ant target to implement this check looks like this:

  <target name="patch.information" description="Gets patch information from code and the database">
    <java
        fork="true"
        classpathref="inttest.classpath"
        failonerror="true" 
        classname="com.tacitknowledge.util.migration.jdbc.MigrationInformation">
      <sysproperty key="migration.systemname" value="${application.name}"/>
    </java>
  </target>

The classpath must contain the migration.properties, the AutoPatch
jar, as well as all of the patch objects, for the information to be
accurate.

For distributed systems, it is important to note that you may not have
two patches with the same patch number in multiple sub-launchers. Each
patch must have a globally unique patch level across all sub-launchers.

4e) Patch Execution
-------------------
Patch execution is relatively straightforward, except for the
possibility that application servers in a cluster may all start at the
same time, and attempt to execute patches at the same time.

In order to ensure that each patch only executes once, the PatchTable
object has logic that locks the patch state table ('patches'),
guaranteeing that multiple servers in a cluster will serialize through
the AutoPatch part of their startup, and thus insuring that the set of 
patches is only executed once.

Other than that, each patch is executed once, with the system patch
level incremented for each successful patch application. Any
unsuccessful patch application will immediately result in voluminous
logging and fatal errors propoaged to the adapter that started AutoPatch, 
as this is a very serious problem and would prevent the correct execution 
of the application, and urgent troubleshooting will usually be required.

Note that it is possible, by setting the "ReadOnly" property, to get AutoPatch
to go through each complete patch phase except for the actual application,
so that you can see what would happen if you applied patches. If the level
isn't correct it will throw an exception when it is read-only. So you can enforce
that the patch level is correct, but not automatically patch, useful in
production configurations.


--------------------------------------------------------------------------


5) How to integrate AutoPatch with a web application
----------------------------------------------------
The easiest way to integrate AutoPatch with a web application is to
add the WebAppMigrationLauncher as a ContextListener for your
application context. This can be done with a web.xml snippet that
looks like this:

    <!-- in the context-params section -->
    <context-param>
        <param-name>migration.systemname</param-name>
        <param-value>mysystemname</param-value>
    </context-param>

    <!-- in the listeners section -->
    <listener>
        <listener-class>com.tacitknowledge.util.migration.jdbc.WebAppMigrationLauncher</listener-class>
    </listener>

"migration.systemname" is used to differentiate different systems
storing their state in the same database (and thus the same patches
table). It also used as a prefix for all migration.properties property
keys.

So long as migration.properties, the AutoPatch library, , its dependent libraries
(from the lib directory of the AutoPatch distribution) and all of
your patches (with their paths specified in migration.properties) are
in the web application's classpath, AutoPatch should work fine.

5b) As of the 0.7 release of autopatch there is a new way to configure your web 
application.  You can use JNDI to supply a data source and eliminate the need for a 
migration.properties file altogether.  Do this by adding the following to your 
web.xml file: 

	<!-- in the context-params section -->
    <context-param>
        <param-name>migration.systemname</param-name>
		<param-value>beekeeper</param-value>
    </context-param>
    <context-param>
        <param-name>migration.readonly</param-name>
		<param-value>false</param-value>
    </context-param>
    <context-param>
        <param-name>migration.databasetype</param-name>
		<param-value>mysql</param-value>
    </context-param>
    <context-param>
        <param-name>migration.patchpath</param-name>
		<param-value>patches:com.stylehive.profile.db.patch</param-value>
    </context-param>
    <context-param>
        <param-name>migration.datasource</param-name>
		<param-value>jdbc/profile</param-value>
    </context-param>
	
	<!-- in the listeners section -->
	<listener>
        <listener-class>com.tacitknowledge.util.migration.jdbc.WebAppJNDIMigrationLauncher</listener-class>
    </listener>

Note that there are similar parameters supplied in the servlet context that once 
existed in the migration.properties file.  All database connection properties have 
been replaced by a parameter called migration.datasource.  This uses an existing 
datasource to perform the autopatch updates.

--------------------------------------------------------------------------


6) How to integrate AutoPatch with your development cycle
---------------------------------------------------------
Its possible to run AutoPatch completely from the command-line, so
that patch development and deployment may take place without need for
a web application container.

The StandaloneMigrationLauncher is used for this, and an ant task to
run this would look like this:

  <target name="patch.database" depends="compile" description="Runs the patch system">
    <java
        fork="true"
        failonerror="true" 
        classname="com.tacitknowledge.util.migration.jdbc.StandaloneMigrationLauncher">
      <classpath refid="inttest.classpath"/>
      <sysproperty key="migration.systemname" value="${application.name}"/>
    </java>
  </target>

If you combine this rule with the patch.information ant target given
above, it should be relatively easy to point at a database, determine
its current patch level in a non-destructive way, and then execute patches.

Some development environments require individual developers to have their own set of
settings so the default 'migration.properties' settings file might not work in this case.
To overcome this issue it is possible to specify an optional 'migration.settings' parameter
(either system property or command line argument) like this:

  <target name="patch.database" depends="compile" description="Runs the patch system">
    <java
        fork="true"
        failonerror="true" 
        classname="com.tacitknowledge.util.migration.jdbc.StandaloneMigrationLauncher">
      <classpath refid="inttest.classpath"/>
      <sysproperty key="migration.systemname" value="${application.name}"/>
      <sysproperty key="migration.settings" value="${username}.properties"/>
    </java>
  </target>

If you happen to rely on command line arguments, please take a note that 'migration.systemname'
must come first (e.g. index 0) followed by 'migration.settings' (e.g. index 1).

--------------------------------------------------------------------------


7) How to implement a patch
---------------------------

7a) SQL data load patch
-----------------------
A SQL data load patch can be implemented by extending
SqlLoadMigrationTask and including the delimited data file...


7b) SQL patch
-------------
SQL patches should be implemented by simply following the SQL file
naming convention outlined above as you create the new SQL patch, 
putting the SQL commands in the patch file, and then by putting the 
patches in the classpath in a location specified in the AutoPatch 
configuration file.


7c) Java patch
--------------
Java patches may be implemented by subclassing MigrationTaskSupport.

7d) XML Data Patch
------------------
Create an xml file that follows the format of DBUnit's FlatXmlDataSet format.
http://dbunit.sourceforge.net/apidocs/org/dbunit/dataset/xml/FlatXmlDataSet.html
Also, it should follow the naming convention patchNNNN_<name>.xml.

--------------------------------------------------------------------------


8) How to extend AutoPatch with new patch types
-----------------------------------------------

8a) Adding New Task Types
-------------------------
By examining SqlScriptMigrationTask, you should be able to get some
ideas on how to extend the system. There's no reason why there
couldn't be an LdapScriptMigrationTask that read .ldif files, or an
XmlScriptMigrationTask that issued SOAP or XML-RPC calls. As long as they
allow themselves to be compared such that they fit into the natural patch
ordering, and errors are propagated out after logging, they will work fine.

8b) Patch Information Persistence
---------------------------------
Currently the patch level of the system is stored in a database, however
the idea of the "system patch level" is a separate subsystem in AutoPatch, and
its not necessary for the patch table to be persisted in a
database. Its entirely possible that a flat file would be sufficient
for some applications, or that LDAP or a web server was the only
external system with persistence.

9) How to rollback patches
---------------------------------
One of the new features in AutoPatch 1.2 is that patches can now be rolled back. 
A rollback is when the patch level of a system is reduced. AutoPatch can execute 
rollback anywhere from one patch to all the patches on a system. All you have to do 
is let AutoPatch know what patch level you'd like to rollback to and then AutoPatch 
will do the rest. 

To execute a rollback, reuse the StandaloneMigrationLauncher and specify a few 
additional parameters.  These are: -rollback to indicate that the desired migration
is a rollback and an integer indicating the desired patch level. So, to run a rollback
from the command line java StandaloneMigrationLauncher -rollback 5.  This command
would instruct AutoPatch to rollback the system to patch level 5. Additionally,
there is a -force parameter.  This parameter tells AutoPatch to ignore a check 
to see if all patches are rollbackable and to force a rollback.  Note that forcing
a rollback is dangergous as it can quite easily cause an exception scenario.

Similar to the patch process, applications that use rollbacks need to instantiate a 
MigrationLauncher. You can accomplish this by implementing an Adapter that executes upon 
application startup or using one of the provided launchers. Instead of calling doMigrations, 
the Launcher should call doRollbacks. This method takes in an int and a boolean. 
The integer indicates the level that the system should rollback to and the boolean 
indicates if the system should attempt to force a rollback. Why would you need to force 
a rollback? Because the people may have implemented their own tasks by implementing the 
MigrationTask interface, it is possible that there are migrations that do not currently 
support rollbacks. Moreover, there are "destructive" migrations that are fundamentally not 
rollbackable. By default, a rollback will only occur if all of the rollback tasks are rollbackable. 
This check happens before any of the rollbacks are executed, so if you attempt to initiate a 
rollback for a set of tasks and one of the tasks is not rollbackable, no rollbacks will execute. 
The logs generated by AutoPatch will indicate which tasks are not rollbackable. To provide the 
greatest level of flexibility, we have included a way of getting around this rule, which is using the 
"-force" flag. This flag tells AutoPatch that it should ignore the rollbackable check and 
proceed with the rollback. Please keep in mind that this is  risky as it's possible to try 
to rollback tasks that throw exceptions if a rollback is attempted. The best option is to 
make your tasks adhere to the new RollbackableMigrationTask interface. If you need to be able 
to force rollbacks, implement the down method to be a no-op method and then also set 
isRollbackSupported to true. Following these steps will allow for AutoPatch to successfully 
execute the task as a rollback.

The important thing to recognize with patch levels is that a rollback will reduce the patch 
level. For example, if you execute a rollback of patch level 5, the patch level of the 
system will be at patch level 4 after the rollback has completed. Rollbacks are executed in 
a descending manner.


The next step is to find the patches. To make this easy, what we've done is deprecated the 
existing MigrationTask object and added a new interface called RollbackableMigrationTask. Additionally, 
we've extended the functionality that loads SQL scripts into tasks to include rollback scripts. 
So, for your existing patches, you'll just need to create a new SQL script and make it available for 
AutoPatch to load. Rollback scripts are named as follows:  patch<patch level>-rollback_<patch name>.sql.  
An example of a valid rollback file name is patch111-rollback_testname.sql. In this case, the 111 is the level 
of the patch. Conceptually, you should think of this script as inherently linked to the forward patch.
The mechanism that associates a patch SQL script and a rollback SQL script is the level. 
So, patch111_testname.sql and patch111-rollback_testname.sql are linked. Please note that the 
name of the patch does not have any role in linking a patch and its rollback. 
In fact, patch111.sql and patch111-rollback_foo.sql would be associated.

Just as with the up process for patching, the ordering in which rollbacks execute is important. 
Autopatch executes rollbacks in a descending manner based upon the patch level. So, if you were to 
issue a rollback to level 10 and the system is at level 13, the patches that rollback would be 13, 12 and 11.
	
10) How to migrate from AutoPatch 1.1 to AutoPatch 1.2
------------------------------------------------------
AutoPatch 1.2 completely supports AutoPatch 1.0, but you'll need to make some changes if you'd like to 
take advantage of the new features.

If you have implemented the MigrationTask interface, you'll need to change the implementation to 
RollbackableMigrationTask. This will require you to implement three new methods. They are up, down and 
isRollbackSupported. The up method performs the exact same purpose as migrate in MigrationTask.  
That is, it performs the actual migration. For simplicity, you can implement up with one line to just 
call migrate. The down method performs a rollback, and the isRollbackSupported method indicates if 
this task can be rolled back. If this method returns false, then the task will not rollback unless the 
force flag is set to true.

If you have extended MigrationTaskSupport, there is actually very little to do. The MigrationTaskSupport 
object already provides a lot of base functionality. By default, MigrationTaskSupport provides the following
functionality. First, the class provides a few default implementations. isRollbackSupported returns the 
value of a boolean variable isRollbackSupported. Note that by default this value is initialized false. 
Secondly, the class provides a base implementation of down, which simply throws UnsupportedOperationException.
The migrate method simply calls up and the up method is a default no op method. To migrate to AutoPatch 1.2, 
simply do the following:  override the up and down methods to perform the migration and rollback and then 
secondly set isRollBackSupported to true if appropriate. Once you've taken these steps, then your task
will be Rollbackable.