Tuning of model file transfer

Owned by Former user (Deleted)
Sept 04, 2017

Tuning of model file transfer (updates) from server to clients and from clients to server.

NOTE: The following is only relevant for clients (EDMappServer or other EDMfatClients) that can directly access the same file system as the actual EDMserver (i.e., clients runningInServerHost). It was experienced that sometimes the file transfer between server and clients is much faster using the Copy File function in windows instead of reading/writing only the necessary updated (changed) blocks. Hence, facility to define what method to use when is implemented. The default method is to always use file transfer by blocks, i.e., only transfer the minimum number of blocks (only changed blocks).

Clients to Server transfer.

Normal operation (block by block transfer):

 

The client sends a message specifying all blocks on ".I" and ".D" files, and database files  (_FILE) to be updated. Each block and _FILE is specified as an EDMLONG. Hence, for very large models the number of bytes to send from client to server can be very big. All existing blocks in the actual ".I" and ".D" to update must be written (by the EDMserver) to a "BIMlog" file before any updates of the servers ".I" and ".D" files. Similar, all new, updated and deleted _FILE files will be renamed to enable rollback/recovery when the actual "clients to server" updates fail. Then all updated blocks can be read from the client and written to the server ".I" and ".D" files. All new and modified _FILE files will be updated using the CopyFile function, i.e, no "partial updates" of _FILE files is possible. When all updates are finished, all BIMlog files and "renamed" _FILE files will be deleted.

 

Using CopyFile function:

The client sends a message specifying all blocks on ".I" and ".D" files, and database files  (_FILE) to be updated. Each block and _FILE is specified as an EDMLONG. Hence, for very large models the number of bytes to send from client to server can be very big. The servers actual ".I" and ".D" files will be renamed (no file transfer) to BIMlog files. Then the actual ".I" and ".D" files will be transferred from client to server using CopyFile function. The handling of _FILE files and the deleting of the BIMlog files will be exactly as for the (block by block transfer).  The EDMconfigurationVariable CLIENTS_TO_SERVER_FILE_TRANSFER can be used to specify when to use "block by block transfer" or "CopyFile transfer". The CLIENTS_TO_SERVER_FILE_TRANSFER value consist of the following two numbers separated by a semicolon

  1. Minimum size in percentage of ".D" file size for using "CopyFile transfer"
  2. Minimum number of blocks to transfer using "CopyFile transfer"

The default value of CLIENTS_TO_SERVER_FILE_TRANSFER is "100 ; 4294967295" meaning always using "block by block transfer". (The "4294967295" number is the largest 32 bits integer.)  Setting the value of CLIENTS_TO_SERVER_FILE_TRANSFER to "50 ; 10000" means: when the volume to transfer is either greater or equal to 50% of the actual '.D' size or greater or equal to 10000 blocks, the 'CopyFile transfer' will be used.

Server to Clients file transfer:

First time a client gets a set of model files ('.I' and '.D' files) from the server, the 'CopyFile transfer' will be used. The transfer of _FILE files from server to client will always use 'CopyFile transfer'. All file transfer between Server and Clients will always be performed by the EDMserver, not by EDMappServer or other EDMfatClients.

Updating a clients existing model files
Normal operation (block by block transfer)

The EDMserver calculates all blocks on the actual '.I' and '.D' files to be updated, and writes these block numbers in the response messaged to the actual Client, one EDMLONG for each block. Also the specification of all deleted _FILE files will be written in the response message, one EDMLONG for each deleted _FILE. Hence, this response message may be quite large for large models. Then the EDMserver will transfer all updated blocks from the server to the client and finally sending the response message to the actual client. The client will use the block specification in the response message to remove all updated blocks from its EDMdbCache.

Using CopyFile function:

 The EDMserver will transfer the actual '.I' and '.D' files using the CopyFile function. For the block number specification, there are two alternatives:

  1. Sending no block number meaning that the actual clients must remove all blocks belonging to the actual '.I' and '.D' files from its EDMdbCache. Hence, only the instanceId of the deleted _FILE files will be sent in the response message, i.e., very small response message.
  2. The sending of block specification, deleted _FILE files and clients EDMdbCache cleaning will be as for 'block by block transfer'.

The EDMconfigurationVariable SERVER_TO_CLIENTS_FILE_TRANSFER can be used to specify when to use 'block by block transfer' or 'CopyFile transfer'. The SERVER_TO_CLIENTS_FILE_TRANSFER value consist of the following three items, two numbers separated by a semicolon and a Boolean value. The two numbers have the same meaning as for the CLIENTS_TO_SERVER_FILE_TRANSFER EDMconfigurationVariable, The boolean value is specifying the following:

  •     TRUE : sending no block specification in response message, i.e., client using full EDMdbCache clear of actual model.
  •     FALSE: sending updated blocks specification in response message (client using partial EDMdbCache clear)

The default value of CLIENTS_TO_SERVER_FILE_TRANSFER is '100 ; 4294967295 ; FALSE' meaning always using 'block by block transfer'. (The '4294967295' number is the largest 32 bits integer.) The boolean value has no meaning for 'block by block transfer'. Setting the value of CLIENTS_TO_SERVER_FILE_TRANSFER to '50 ; 10000 ; TRUE' means: when the volume to transfer is either greater or equal to 50% of the actual '.D' size or greater or equal to 10000 blocks, the 'CopyFile transfer' will be used. No block specification will be sent in the response message from EDMserver to the actual client. The client must clear all the entries for the actual model from its EDMdbCache. As the threshold for using 'block by block transfer' or 'CopyFile transfer' may vary from system to system, it is difficult to make this an automatic process or to give good advice. We plan to make a 'parameterized' EDMsupervisor function that can run both methods and give measured 'response times' to give indications where the threshold between 'block by block transfer' and 'CopyFile transfer' on a particular system.