integrate in this scheme the listening functionalities, and so forth. The core of this
solution is the use of condition variables to manage synchronization. As one can
see in Figure 7, pseudocode, the synchronization scheme, is quite simple.
This mechanism gives one complete parallelism in reading/elab/writing (read-ahead
and write-behind) using simple read() and write() functions.
This approach gives a clear definition of the data flow, where these come from,
where their transformation code resides, and what their target is. All the transformation
code is contained in a processUnit (???elab()??? in the schema above) where
the incoming data, at application level, are viewed record by record, but internally
processed block by block for performance reasons.
We have spoken about a ???hub??? because this architecture with several buffer pools,
several groups of specialized threads (for reading from file or another database,
for writing in direct path or SQL or to file, etc.), several groups of transformation
threads (in which application code resides) is well suited to be viewed as a hub, in
which many flows converge and other data flows are delivered to different DBMS
tables or other destinations.
Operation.and.Maintenance.Issues
One of the most critical aspects in a DW project is ???operation and maintenance???
(O&M). Even the best ETL application can fail for many reasons, sometimes due
Figure 7.
Pages:
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221