Field3D
SparseFileManager Class Reference

#include <SparseFile.h>

Public Types

typedef std::list< SparseFile::CacheBlockCacheList
 

Public Member Functions

template<class Data_T >
void activateBlock (int fileId, int blockIdx)
 Called by SparseField when it's about to read from a block. This should not be called by the user, and may be removed from the public interface later. More...
 
float cacheEfficiency ()
 Computes the efficiency, the ratio of the number of blocks ever loaded to the number of loads. If this is <1, then there were reloads. More...
 
float cacheFractionLoaded ()
 Computes the ratio of blocks in the cache to the total number of blocks that have been loaded (including unloaded blocks) More...
 
float cacheLoadsPerBlock ()
 Computes the overall loaded-blocks-to-load ratio for cached files. More...
 
template<class Data_T >
void decBlockRef (int fileId, int blockIdx)
 Decrements the usage reference count on the specified block, after its value is no longer being used This should not be called by the user, and may be removed from the public interface later. More...
 
bool doLimitMemUse () const
 Returns whether to limit memory usage and do dynamic loading for sparse fields. More...
 
void flushCache ()
 Flushes the entire block cache for all files, should probably only be used for debugging. More...
 
template<class Data_T >
void incBlockRef (int fileId, int blockIdx)
 Increments the usage reference count on the specified block, to prevent it from getting unloaded while it's still in use. This should not be called by the user, and may be removed from the public interface later. More...
 
long long int memSize () const
 Returns the number of bytes used by the SparseFileManager itself. More...
 
long long numLoadedBlocks ()
 Returns the total number of blocks currently loaded into cache. More...
 
void resetCacheStatistics ()
 Resets block load. More...
 
void setLimitMemUse (bool enabled)
 Sets whether to limit memory usage and do dynamic loading for sparse fields. More...
 
void setMaxMemUse (float maxMemUse)
 Sets the maximum memory usage, in MB, by dynamically loaded sparse fields. More...
 
long long totalLoadedBlocks ()
 Returns the total number of blocks loaded (max 1 per block) into cache. More...
 
long long totalLoads ()
 Returns the total number of block loads in the cache. More...
 

Static Public Member Functions

static SparseFileManagersingleton ()
 Returns a reference to the singleton instance. More...
 

Protected Member Functions

template<class Data_T >
int getNextId (const std::string filename, const std::string layerPath)
 Returns the id of the next cache item. This is stored in the SparseField in order to reference its fields at a later time. More...
 
template<class Data_T >
SparseFile::Reference< Data_T > * reference (int index)
 Returns a reference to the Reference object with the given index. More...
 
template<class Data_T >
void removeFieldFromCache (int refIdx)
 

Private Member Functions

void addBlockToCache (DataTypeEnum blockType, int fileId, int blockIdx)
 Adds the newly loaded block to the cache, managed by the paging algorithm. More...
 
template<class Data_T >
void deallocateBlock (CacheList::iterator &it)
 Utility function to deallocate a single block. More...
 
template<class Data_T >
int64_t deallocateBlock (const SparseFile::CacheBlock &cb)
 Utility function to attempt to deallocate a single block and advance the "hand". More...
 
void deallocateBlocks (int64_t bytesNeeded)
 Utility function to reclaim the specified number of bytes by deallocating unneeded blocks. More...
 
 SparseFileManager ()
 Private to prevent instantiation. More...
 

Private Attributes

CacheList m_blockCacheList
 List of dynamically loaded blocks to be considered for unloading when the cache is full. Currently using Second-chance/Clock paging algorithm. For a description of the algorithm, look at: http://en.wikipedia.org/wiki/Page_replacement_algorithm#Second-chance. More...
 
SparseFile::FileReferences m_fileData
 Vector containing information for each of the managed fields. The order matches the index stored in each SparseField::m_fileId. More...
 
bool m_limitMemUse
 Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when true. More...
 
float m_maxMemUse
 Max amount om memory to use in megabytes. More...
 
int64_t m_maxMemUseInBytes
 Max amount om memory to use in bytes. More...
 
int64_t m_memUse
 Current amount of memory in use in bytes. More...
 
boost::mutex m_mutex
 Mutex to prevent multiple threads from deallocating blocks at the same time. More...
 
CacheList::iterator m_nextBlock
 Pointer to the next block to test for unloading in the cache, the "hand" of the clock. More...
 

Static Private Attributes

static boost::scoped_ptr< SparseFileManagerms_singleton
 Pointer to singleton. More...
 

Friends

template<class Data_T >
class SparseField
 

Detailed Description

Handles specifics about reading sparse fields from disk. Its primary use is to control sparse fields read using memory limiting (dynamic loading).

To enable the dynamic cache for a file, call setLimitMemUse(true) before opening the file. If you want other files to be fully loaded, call setLimitMemUse(false).

Example of how to use the cache manager to automatically unload sparse blocks from a f3d file:

SparseFileManager &sparseManager = SparseFileManager::singleton();
sparseManager.setLimitMemUse(true);  // enables cache for files to be opened 
sparseManager.setMaxMemUse(1000.0);  // sets cache to 1 GB
Field3DInputFile cacheManagedFile;
if (!cacheManagedFile.open(filename)) {
  Msg::print( "Couldn't open file: " + filename);
  return 1;
}
sparseManager.setLimitMemUse(false);  // disables cache for other files
... // You can use the file normally, loading layers and then accessing
... // with const_iterator, value(), or empty block functions like
... // getBlockEmptyValue().
... // Layers loaded from cacheManagedFile will be managed by the cache,
... // but other files you open will be fully loaded when opened because of
... // the setLimitMemUse(false) call.
Msg::print("blocks in cache: " +
  boost::lexical_cast<std::string>(sparseManager.numLoadedBlocks()));
Msg::print("cache blocks ever loaded: " +
  boost::lexical_cast<std::string>(sparseManager.totalLoadedBlocks()));
Msg::print("cache loads: " +
  boost::lexical_cast<std::string>(sparseManager.totalLoads()));
Msg::print("cache fraction loaded: " +
  boost::lexical_cast<std::string>(sparseManager.cacheFractionLoaded()));
Msg::print("cache loads per block: " +
  boost::lexical_cast<std::string>(sparseManager.cacheLoadsPerBlock()));
Msg::print("cache efficiency: " +
  boost::lexical_cast<std::string>(sparseManager.cacheEfficiency()));

If you want to flush the cache manually instead of waiting for the process to end and clean up its memory:

sparseManager.flushCache();
sparseManager.resetCacheStatistics();

Definition at line 397 of file SparseFile.h.

Member Typedef Documentation

◆ CacheList

Definition at line 407 of file SparseFile.h.

Constructor & Destructor Documentation

◆ SparseFileManager()

SparseFileManager::SparseFileManager ( )
private

Private to prevent instantiation.

Definition at line 282 of file SparseFile.cpp.

283  : m_memUse(0),
284  m_limitMemUse(false)
285 {
286  setMaxMemUse(1000.0);
287  m_nextBlock = m_blockCacheList.begin();
288 }

References m_blockCacheList, m_nextBlock, and setMaxMemUse().

Member Function Documentation

◆ singleton()

SparseFileManager & SparseFileManager::singleton ( )
static

Returns a reference to the singleton instance.

Definition at line 66 of file SparseFile.cpp.

67 {
68  if (ms_singleton.get() == NULL) {
70  }
71  return *ms_singleton;
72 }

References ms_singleton.

Referenced by SparseField< Data_T >::addReference().

◆ setLimitMemUse()

void SparseFileManager::setLimitMemUse ( bool  enabled)

Sets whether to limit memory usage and do dynamic loading for sparse fields.

Definition at line 76 of file SparseFile.cpp.

77 {
78  m_limitMemUse = enabled;
79 }

References m_limitMemUse.

◆ doLimitMemUse()

bool SparseFileManager::doLimitMemUse ( ) const

Returns whether to limit memory usage and do dynamic loading for sparse fields.

Definition at line 83 of file SparseFile.cpp.

84 {
85  return m_limitMemUse;
86 }

References m_limitMemUse.

◆ setMaxMemUse()

void SparseFileManager::setMaxMemUse ( float  maxMemUse)

Sets the maximum memory usage, in MB, by dynamically loaded sparse fields.

Definition at line 90 of file SparseFile.cpp.

91 {
92  m_maxMemUse = maxMemUse;
93  m_maxMemUseInBytes = static_cast<int64_t>(m_maxMemUse * 1024 * 1024);
94 }

References m_maxMemUse, and m_maxMemUseInBytes.

Referenced by SparseFileManager().

◆ flushCache()

void SparseFileManager::flushCache ( )

Flushes the entire block cache for all files, should probably only be used for debugging.

Definition at line 223 of file SparseFile.cpp.

224 {
225  boost::mutex::scoped_lock lock(m_mutex);
226 
227  CacheList::iterator it = m_blockCacheList.begin();
228  while (it != m_blockCacheList.end()) {
229  SparseFile::CacheBlock &cb = *it;
230 
231  switch(cb.blockType) {
232  case DataTypeHalf:
233  deallocateBlock<half>(it);
234  break;
235  case DataTypeFloat:
236  deallocateBlock<float>(it);
237  break;
238  case DataTypeDouble:
239  deallocateBlock<double>(it);
240  break;
241  case DataTypeVecHalf:
242  deallocateBlock<V3h>(it);
243  break;
244  case DataTypeVecFloat:
245  deallocateBlock<V3f>(it);
246  break;
247  case DataTypeVecDouble:
248  deallocateBlock<V3d>(it);
249  break;
250  case DataTypeUnknown:
251  default:
252  break;
253  }
254  }
255  m_nextBlock = m_blockCacheList.begin();
256 }

References SparseFile::CacheBlock::blockType, DataTypeDouble, DataTypeFloat, DataTypeHalf, DataTypeUnknown, DataTypeVecDouble, DataTypeVecFloat, DataTypeVecHalf, m_blockCacheList, m_mutex, and m_nextBlock.

◆ totalLoads()

long long SparseFileManager::totalLoads ( )

Returns the total number of block loads in the cache.

Definition at line 292 of file SparseFile.cpp.

293 {
294 
295  long long int numLoads = 0;
296 
297  for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
298  numLoads += m_fileData.ref<half>(i)->totalLoads();
299  }
300 
301  for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
302  numLoads += m_fileData.ref<V3h>(i)->totalLoads();
303  }
304 
305  for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
306  numLoads += m_fileData.ref<float>(i)->totalLoads();
307  }
308 
309  for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
310  numLoads += m_fileData.ref<V3f>(i)->totalLoads();
311  }
312 
313  for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
314  numLoads += m_fileData.ref<double>(i)->totalLoads();
315  }
316 
317  for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
318  numLoads += m_fileData.ref<V3d>(i)->totalLoads();
319  }
320  return numLoads;
321 }

References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().

Referenced by cacheEfficiency(), and cacheLoadsPerBlock().

◆ numLoadedBlocks()

long long SparseFileManager::numLoadedBlocks ( )

Returns the total number of blocks currently loaded into cache.

Definition at line 325 of file SparseFile.cpp.

326 {
327 
328  long long int numBlocks = 0;
329 
330  for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
331  numBlocks += m_fileData.ref<half>(i)->numLoadedBlocks();
332  }
333 
334  for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
335  numBlocks += m_fileData.ref<V3h>(i)->numLoadedBlocks();
336  }
337 
338  for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
339  numBlocks += m_fileData.ref<float>(i)->numLoadedBlocks();
340  }
341 
342  for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
343  numBlocks += m_fileData.ref<V3f>(i)->numLoadedBlocks();
344  }
345 
346  for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
347  numBlocks += m_fileData.ref<double>(i)->numLoadedBlocks();
348  }
349 
350  for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
351  numBlocks += m_fileData.ref<V3d>(i)->numLoadedBlocks();
352  }
353  return numBlocks;
354 }

References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().

Referenced by cacheFractionLoaded().

◆ totalLoadedBlocks()

long long SparseFileManager::totalLoadedBlocks ( )

Returns the total number of blocks loaded (max 1 per block) into cache.

Definition at line 358 of file SparseFile.cpp.

359 {
360 
361  long long int numBlocks = 0;
362 
363  for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
364  numBlocks += m_fileData.ref<half>(i)->totalLoadedBlocks();
365  }
366 
367  for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
368  numBlocks += m_fileData.ref<V3h>(i)->totalLoadedBlocks();
369  }
370 
371  for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
372  numBlocks += m_fileData.ref<float>(i)->totalLoadedBlocks();
373  }
374 
375  for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
376  numBlocks += m_fileData.ref<V3f>(i)->totalLoadedBlocks();
377  }
378 
379  for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
380  numBlocks += m_fileData.ref<double>(i)->totalLoadedBlocks();
381  }
382 
383  for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
384  numBlocks += m_fileData.ref<V3d>(i)->totalLoadedBlocks();
385  }
386  return numBlocks;
387 }

References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().

Referenced by cacheEfficiency(), cacheFractionLoaded(), and cacheLoadsPerBlock().

◆ cacheFractionLoaded()

float SparseFileManager::cacheFractionLoaded ( )

Computes the ratio of blocks in the cache to the total number of blocks that have been loaded (including unloaded blocks)

Definition at line 391 of file SparseFile.cpp.

392 {
393  return ((double)numLoadedBlocks())/std::max(1.0, ((double)totalLoadedBlocks()));
394 }

References detail::max(), numLoadedBlocks(), and totalLoadedBlocks().

◆ cacheLoadsPerBlock()

float SparseFileManager::cacheLoadsPerBlock ( )

Computes the overall loaded-blocks-to-load ratio for cached files.

Definition at line 398 of file SparseFile.cpp.

399 {
400  return ((double)totalLoads())/std::max(1.0, ((double)totalLoadedBlocks()));
401 }

References detail::max(), totalLoadedBlocks(), and totalLoads().

◆ cacheEfficiency()

float SparseFileManager::cacheEfficiency ( )

Computes the efficiency, the ratio of the number of blocks ever loaded to the number of loads. If this is <1, then there were reloads.

Definition at line 405 of file SparseFile.cpp.

406 {
407  return ((double)totalLoadedBlocks())/std::max(1.0, ((double)totalLoads()));
408 }

References detail::max(), totalLoadedBlocks(), and totalLoads().

◆ resetCacheStatistics()

void SparseFileManager::resetCacheStatistics ( )

Resets block load.

Definition at line 412 of file SparseFile.cpp.

413 {
414 
415  for (size_t i=0; i<m_fileData.numRefs<half>(); i++) {
417  }
418 
419  for (size_t i=0; i<m_fileData.numRefs<V3h>(); i++) {
421  }
422 
423  for (size_t i=0; i<m_fileData.numRefs<float>(); i++) {
424  m_fileData.ref<float>(i)->resetCacheStatistics();
425  }
426 
427  for (size_t i=0; i<m_fileData.numRefs<V3f>(); i++) {
429  }
430 
431  for (size_t i=0; i<m_fileData.numRefs<double>(); i++) {
432  m_fileData.ref<double>(i)->resetCacheStatistics();
433  }
434 
435  for (size_t i=0; i<m_fileData.numRefs<V3d>(); i++) {
437  }
438 }

References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().

◆ memSize()

long long int SparseFileManager::memSize ( ) const

Returns the number of bytes used by the SparseFileManager itself.

Definition at line 442 of file SparseFile.cpp.

443 {
444  boost::mutex::scoped_lock lock(m_mutex);
445 
446  return sizeof(*this) + m_fileData.memSize() +
447  m_blockCacheList.size() * sizeof(SparseFile::CacheBlock);
448 }

References m_blockCacheList, m_fileData, m_mutex, and SparseFile::FileReferences::memSize().

◆ incBlockRef()

template<class Data_T >
void SparseFileManager::incBlockRef ( int  fileId,
int  blockIdx 
)

Increments the usage reference count on the specified block, to prevent it from getting unloaded while it's still in use. This should not be called by the user, and may be removed from the public interface later.

Definition at line 1298 of file SparseFile.h.

1299 {
1301 
1302  if (reference->fileBlockIndices[blockIdx] >= 0) {
1303  reference->incBlockRef(blockIdx);
1304  }
1305 }

References m_fileData, SparseFile::FileReferences::ref(), and reference().

◆ decBlockRef()

template<class Data_T >
void SparseFileManager::decBlockRef ( int  fileId,
int  blockIdx 
)

Decrements the usage reference count on the specified block, after its value is no longer being used This should not be called by the user, and may be removed from the public interface later.

Definition at line 1311 of file SparseFile.h.

1312 {
1314 
1315  if (reference->fileBlockIndices[blockIdx] >= 0) {
1316  reference->decBlockRef(blockIdx);
1317  }
1318 }

References m_fileData, SparseFile::FileReferences::ref(), and reference().

◆ activateBlock()

template<class Data_T >
void SparseFileManager::activateBlock ( int  fileId,
int  blockIdx 
)

Called by SparseField when it's about to read from a block. This should not be called by the user, and may be removed from the public interface later.

Definition at line 1257 of file SparseFile.h.

1258 {
1260 
1261  if (reference->fileBlockIndices[blockIdx] >= 0) {
1262  if (!reference->blockLoaded[blockIdx]) {
1263  int blockSize = reference->blockSize(blockIdx);
1264  if (m_limitMemUse) {
1265  // if we already have enough free memory, deallocateBlocks()
1266  // will just return
1267  deallocateBlocks(blockSize);
1268  }
1269 
1270  if (!reference->fileIsOpen()) {
1271  reference->openFile();
1272  }
1273 
1274  boost::mutex::scoped_lock lock_A(m_mutex);
1275 #if F3D_SHORT_MUTEX_ARRAY
1276  boost::mutex::scoped_lock
1277  lock_B(reference->blockMutex[blockIdx % reference->blockMutexSize]);
1278 #else
1279  boost::mutex::scoped_lock lock_B(reference->blockMutex[blockIdx]);
1280 #endif
1281  // check to see if it was loaded between when the function
1282  // started and we got the lock on the block
1283  if (!reference->blockLoaded[blockIdx]) {
1284  reference->loadBlock(blockIdx);
1285  reference->loadCounts[blockIdx]++;
1287  m_memUse += blockSize;
1288  }
1289  }
1290  }
1291  reference->blockUsed[blockIdx] = true;
1292 }

References addBlockToCache(), deallocateBlocks(), m_fileData, m_limitMemUse, m_memUse, m_mutex, SparseFile::FileReferences::ref(), and reference().

Referenced by SparseField< Data_T >::const_iterator::operator->().

◆ reference()

template<class Data_T >
SparseFile::Reference< Data_T > * SparseFileManager::reference ( int  index)
protected

Returns a reference to the Reference object with the given index.

Definition at line 1248 of file SparseFile.h.

1249 {
1250  return m_fileData.ref<Data_T>(index);
1251 }

References m_fileData, and SparseFile::FileReferences::ref().

Referenced by activateBlock(), deallocateBlock(), decBlockRef(), incBlockRef(), and removeFieldFromCache().

◆ getNextId()

template<class Data_T >
int SparseFileManager::getNextId ( const std::string  filename,
const std::string  layerPath 
)
protected

Returns the id of the next cache item. This is stored in the SparseField in order to reference its fields at a later time.

Definition at line 1182 of file SparseFile.h.

1184 {
1185  using namespace SparseFile;
1186 
1187  // Must hold a mutex while appending to m_fileData
1188  boost::mutex::scoped_lock lock(m_mutex);
1189 
1190  int id = m_fileData.append<Data_T>(Reference<Data_T>::create(filename,
1191  layerPath));
1192  return id;
1193 }

References SparseFile::FileReferences::append(), m_fileData, and m_mutex.

◆ removeFieldFromCache()

template<class Data_T >
void SparseFileManager::removeFieldFromCache ( int  refIdx)
protected

Definition at line 1199 of file SparseFile.h.

1200 {
1201  boost::mutex::scoped_lock lock(m_mutex);
1202 
1205 
1206  CacheList::iterator it = m_blockCacheList.begin();
1207  CacheList::iterator end = m_blockCacheList.end();
1208  CacheList::iterator next;
1209 
1210  int64_t bytesFreed = 0;
1211 
1212  while (it != end) {
1213  if (it->blockType == blockType && it->refIdx == refIdx) {
1214  if (it == m_nextBlock) {
1215  ++m_nextBlock;
1216  }
1217  next = it;
1218  ++next;
1219  bytesFreed += reference->blockSize(it->blockIdx);
1220  m_blockCacheList.erase(it);
1221  it = next;
1222  } else {
1223  ++it;
1224  }
1225  }
1226  m_memUse -= bytesFreed;
1227 
1228  std::vector<int>().swap(reference->fileBlockIndices);
1229 #if F3D_NO_BLOCKS_ARRAY
1230  reference->fileBlockIndices.resize(reference->numBlocks, -1);
1231 #else
1232  reference->fileBlockIndices.resize(reference->blocks.size(), -1);
1233  typedef typename SparseFile::Reference<Data_T>::BlockPtrs BlockPtrs;
1234  BlockPtrs().swap(reference->blocks);
1235 #endif
1236  std::vector<int>().swap(reference->blockLoaded);
1237  std::vector<bool>().swap(reference->blockUsed);
1238  std::vector<int>().swap(reference->loadCounts);
1239  std::vector<int>().swap(reference->refCounts);
1240  delete[] reference->blockMutex;
1241  reference->blockMutex = NULL;
1242 }

References m_blockCacheList, m_fileData, m_memUse, m_mutex, m_nextBlock, SparseFile::FileReferences::ref(), reference(), and DataTypeTraits< T >::typeEnum().

◆ addBlockToCache()

void SparseFileManager::addBlockToCache ( DataTypeEnum  blockType,
int  fileId,
int  blockIdx 
)
private

Adds the newly loaded block to the cache, managed by the paging algorithm.

Definition at line 260 of file SparseFile.cpp.

262 {
263  // Note: this lock is obtained while we also have a lock on the
264  // specific block (in activateBlock()), so we should make sure we
265  // never lock the SparseFileManager and *then* a block, to ensure we
266  // don't have a deadlock.
267  //
268  // Note: this was changed so the order was consistent w/ dealloc
269  // again, see activateBlock()
270  // boost::mutex::scoped_lock lock(m_mutex);
271 
272  SparseFile::CacheBlock block(blockType, fileId, blockIdx);
273  if (m_nextBlock == m_blockCacheList.end()) {
274  m_blockCacheList.push_back(block);
275  } else {
276  m_blockCacheList.insert(m_nextBlock, block);
277  }
278 }

References m_blockCacheList, and m_nextBlock.

Referenced by activateBlock().

◆ deallocateBlocks()

void SparseFileManager::deallocateBlocks ( int64_t  bytesNeeded)
private

Utility function to reclaim the specified number of bytes by deallocating unneeded blocks.

Definition at line 160 of file SparseFile.cpp.

161 {
162  boost::mutex::scoped_lock lock_A(m_mutex);
163 
164  while (m_blockCacheList.begin() != m_blockCacheList.end() &&
165  m_maxMemUseInBytes-m_memUse < bytesNeeded) {
166 
167  if (m_nextBlock == m_blockCacheList.end())
168  m_nextBlock = m_blockCacheList.begin();
169 
171 
172  // if bytesFreed is set to >0, then we've already freed a block
173  // and advanced the "clock hand" iterator
174  int64_t bytesFreed = 0;
175 
176  switch(cb.blockType) {
177  case DataTypeHalf:
178  bytesFreed = deallocateBlock<half>(cb);
179  if (bytesFreed > 0) {
180  continue;
181  }
182  break;
183  case DataTypeFloat:
184  bytesFreed = deallocateBlock<float>(cb);
185  if (bytesFreed > 0) {
186  continue;
187  }
188  break;
189  case DataTypeDouble:
190  bytesFreed = deallocateBlock<double>(cb);
191  if (bytesFreed > 0) {
192  continue;
193  }
194  break;
195  case DataTypeVecHalf:
196  bytesFreed = deallocateBlock<V3h>(cb);
197  if (bytesFreed > 0) {
198  continue;
199  }
200  break;
201  case DataTypeVecFloat:
202  bytesFreed = deallocateBlock<V3f>(cb);
203  if (bytesFreed > 0) {
204  continue;
205  }
206  break;
207  case DataTypeVecDouble:
208  bytesFreed = deallocateBlock<V3d>(cb);
209  if (bytesFreed > 0) {
210  continue;
211  }
212  break;
213  case DataTypeUnknown:
214  default:
215  break;
216  }
217  ++m_nextBlock;
218  }
219 }

References SparseFile::CacheBlock::blockType, DataTypeDouble, DataTypeFloat, DataTypeHalf, DataTypeUnknown, DataTypeVecDouble, DataTypeVecFloat, DataTypeVecHalf, m_blockCacheList, m_maxMemUseInBytes, m_memUse, m_mutex, and m_nextBlock.

Referenced by activateBlock().

◆ deallocateBlock() [1/2]

template<class Data_T >
int64_t SparseFileManager::deallocateBlock ( const SparseFile::CacheBlock cb)
private

Utility function to attempt to deallocate a single block and advance the "hand".

Definition at line 99 of file SparseFile.cpp.

100 {
101  int64_t bytesFreed = 0;
103 
104  // Note: we don't need to lock the block's mutex because
105  // deallocateBlock() is only called while the SparseFileManager's
106  // mutex is also locked (in flushCache() or deallocateBlocks()).
107  // Don't lock the block, to make sure we don't have a deadlock by
108  // holding two locks at the same time. (Because addBlockToCache()
109  // locks the manager but is also in a block-specific lock.)
110 
111  // lock the current block to make sure its blockUsed flag and ref
112  // counts don't change
113  // Note: this lock order is made consistent w/ allocate to prevent
114  // deadlocks and crashes.
115 
116 #if F3D_SHORT_MUTEX_ARRAY
117  boost::mutex::scoped_lock
118  lock_B(reference->blockMutex[cb.blockIdx % reference->blockMutexSize]);
119 #else
120  boost::mutex::scoped_lock lock_B(reference->blockMutex[cb.blockIdx]);
121 #endif
122 
123  // check whether the block is still in use
124  if (reference->refCounts[cb.blockIdx] > 0)
125  return bytesFreed;
126 
127  if (reference->blockUsed[cb.blockIdx]) {
128  // the block was recently used according to Second-chance paging
129  // algorithm, so skip it
130  reference->blockUsed[cb.blockIdx] = false;
131  }
132  else {
133 
134  // the block wasn't in use, so free it
135  reference->unloadBlock(cb.blockIdx);
136  bytesFreed = reference->blockSize(cb.blockIdx);
137  m_memUse -= bytesFreed;
138  CacheList::iterator toRemove = m_nextBlock;
139  ++m_nextBlock;
140  m_blockCacheList.erase(toRemove);
141  }
142  return bytesFreed;
143 }

References SparseFile::CacheBlock::blockIdx, m_blockCacheList, m_fileData, m_memUse, m_nextBlock, SparseFile::FileReferences::ref(), reference(), and SparseFile::CacheBlock::refIdx.

◆ deallocateBlock() [2/2]

template<class Data_T >
void SparseFileManager::deallocateBlock ( CacheList::iterator &  it)
private

Utility function to deallocate a single block.

Definition at line 148 of file SparseFile.cpp.

149 {
150  SparseFile::CacheBlock &cb = *it;
152  int64_t bytesFreed = reference->blockSize(cb.blockIdx);
153  m_memUse -= bytesFreed;
154  reference->unloadBlock(cb.blockIdx);
155  it = m_blockCacheList.erase(it);
156 }

References SparseFile::CacheBlock::blockIdx, m_blockCacheList, m_fileData, m_memUse, SparseFile::FileReferences::ref(), reference(), and SparseFile::CacheBlock::refIdx.

Friends And Related Function Documentation

◆ SparseField

template<class Data_T >
friend class SparseField
friend

Definition at line 403 of file SparseFile.h.

Member Data Documentation

◆ ms_singleton

FIELD3D_NAMESPACE_OPEN boost::scoped_ptr< SparseFileManager > SparseFileManager::ms_singleton
staticprivate

Pointer to singleton.

Definition at line 498 of file SparseFile.h.

Referenced by singleton().

◆ m_maxMemUse

float SparseFileManager::m_maxMemUse
private

Max amount om memory to use in megabytes.

Definition at line 517 of file SparseFile.h.

Referenced by setMaxMemUse().

◆ m_maxMemUseInBytes

int64_t SparseFileManager::m_maxMemUseInBytes
private

Max amount om memory to use in bytes.

Definition at line 520 of file SparseFile.h.

Referenced by deallocateBlocks(), and setMaxMemUse().

◆ m_memUse

int64_t SparseFileManager::m_memUse
private

Current amount of memory in use in bytes.

Definition at line 523 of file SparseFile.h.

Referenced by activateBlock(), deallocateBlock(), deallocateBlocks(), and removeFieldFromCache().

◆ m_limitMemUse

bool SparseFileManager::m_limitMemUse
private

Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when true.

Definition at line 527 of file SparseFile.h.

Referenced by activateBlock(), doLimitMemUse(), and setLimitMemUse().

◆ m_fileData

SparseFile::FileReferences SparseFileManager::m_fileData
private

Vector containing information for each of the managed fields. The order matches the index stored in each SparseField::m_fileId.

Definition at line 531 of file SparseFile.h.

Referenced by activateBlock(), deallocateBlock(), decBlockRef(), getNextId(), incBlockRef(), memSize(), numLoadedBlocks(), reference(), removeFieldFromCache(), resetCacheStatistics(), totalLoadedBlocks(), and totalLoads().

◆ m_blockCacheList

CacheList SparseFileManager::m_blockCacheList
private

List of dynamically loaded blocks to be considered for unloading when the cache is full. Currently using Second-chance/Clock paging algorithm. For a description of the algorithm, look at: http://en.wikipedia.org/wiki/Page_replacement_algorithm#Second-chance.

Definition at line 538 of file SparseFile.h.

Referenced by addBlockToCache(), deallocateBlock(), deallocateBlocks(), flushCache(), memSize(), removeFieldFromCache(), and SparseFileManager().

◆ m_nextBlock

CacheList::iterator SparseFileManager::m_nextBlock
private

Pointer to the next block to test for unloading in the cache, the "hand" of the clock.

Definition at line 542 of file SparseFile.h.

Referenced by addBlockToCache(), deallocateBlock(), deallocateBlocks(), flushCache(), removeFieldFromCache(), and SparseFileManager().

◆ m_mutex

boost::mutex SparseFileManager::m_mutex
mutableprivate

Mutex to prevent multiple threads from deallocating blocks at the same time.

Definition at line 546 of file SparseFile.h.

Referenced by activateBlock(), deallocateBlocks(), flushCache(), getNextId(), memSize(), and removeFieldFromCache().


The documentation for this class was generated from the following files:
SparseFileManager::reference
SparseFile::Reference< Data_T > * reference(int index)
Returns a reference to the Reference object with the given index.
Definition: SparseFile.h:1248
SparseFileManager::setMaxMemUse
void setMaxMemUse(float maxMemUse)
Sets the maximum memory usage, in MB, by dynamically loaded sparse fields.
Definition: SparseFile.cpp:90
SparseFile::CacheBlock::refIdx
int refIdx
Definition: SparseFile.h:325
V3d
Imath::V3d V3d
Definition: SpiMathLib.h:74
SparseFile::FileReferences::numRefs
size_t numRefs() const
Returns the number of file references of the corresponding collection.
Sparse::SparseBlock
Definition: SparseField.h:228
SparseFileManager::m_blockCacheList
CacheList m_blockCacheList
List of dynamically loaded blocks to be considered for unloading when the cache is full....
Definition: SparseFile.h:538
SparseFileManager::totalLoadedBlocks
long long totalLoadedBlocks()
Returns the total number of blocks loaded (max 1 per block) into cache.
Definition: SparseFile.cpp:358
DataTypeFloat
@ DataTypeFloat
Definition: Traits.h:112
DataTypeEnum
DataTypeEnum
Definition: Traits.h:108
DataTypeUnknown
@ DataTypeUnknown
Definition: Traits.h:117
V3f
Imath::V3f V3f
Definition: SpiMathLib.h:73
SparseFile
Definition: SparseFile.h:91
SparseFileManager::m_maxMemUseInBytes
int64_t m_maxMemUseInBytes
Max amount om memory to use in bytes.
Definition: SparseFile.h:520
SparseFile::FileReferences::ref
Reference< Data_T > * ref(size_t idx)
Returns a reference to the index. This is specialized so that the correct data member is accessed.
DataTypeVecHalf
@ DataTypeVecHalf
Definition: Traits.h:114
SparseFile::CacheBlock::blockIdx
int blockIdx
Definition: SparseFile.h:326
SparseFileManager::m_limitMemUse
bool m_limitMemUse
Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when tr...
Definition: SparseFile.h:527
SparseFile::FileReferences::append
size_t append(typename Reference< Data_T >::Ptr ref)
Appends a reference to the collection. This is specialized so that the correct data member is accesse...
DataTypeHalf
@ DataTypeHalf
Definition: Traits.h:109
half
FIELD3D_NAMESPACE_OPEN typedef ::half half
Definition: SpiMathLib.h:64
SparseFile::CacheBlock::blockType
DataTypeEnum blockType
Definition: SparseFile.h:324
SparseFileManager::m_fileData
SparseFile::FileReferences m_fileData
Vector containing information for each of the managed fields. The order matches the index stored in e...
Definition: SparseFile.h:531
SparseFileManager::m_memUse
int64_t m_memUse
Current amount of memory in use in bytes.
Definition: SparseFile.h:523
detail::max
T max(const T a, const T2 b)
Max operation on mixed types.
Definition: FieldSampler.h:32
SparseFileManager::m_maxMemUse
float m_maxMemUse
Max amount om memory to use in megabytes.
Definition: SparseFile.h:517
SparseFileManager::m_mutex
boost::mutex m_mutex
Mutex to prevent multiple threads from deallocating blocks at the same time.
Definition: SparseFile.h:546
DataTypeTraits::typeEnum
static DataTypeEnum typeEnum()
SparseFileManager::addBlockToCache
void addBlockToCache(DataTypeEnum blockType, int fileId, int blockIdx)
Adds the newly loaded block to the cache, managed by the paging algorithm.
Definition: SparseFile.cpp:260
SparseFileManager
Definition: SparseFile.h:398
DataTypeVecDouble
@ DataTypeVecDouble
Definition: Traits.h:116
DataTypeVecFloat
@ DataTypeVecFloat
Definition: Traits.h:115
SparseFileManager::numLoadedBlocks
long long numLoadedBlocks()
Returns the total number of blocks currently loaded into cache.
Definition: SparseFile.cpp:325
SparseFileManager::m_nextBlock
CacheList::iterator m_nextBlock
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition: SparseFile.h:542
SparseFile::Reference
Definition: SparseFile.h:106
DataTypeDouble
@ DataTypeDouble
Definition: Traits.h:113
V3h
Imath::Vec3< half > V3h
Definition: SpiMathLib.h:72
SparseFile::FileReferences::memSize
long long int memSize() const
Returns the memory use for the refs.
Definition: SparseFile.cpp:452
DataTypeTraits
Definition: Traits.h:266
SparseFileManager::deallocateBlocks
void deallocateBlocks(int64_t bytesNeeded)
Utility function to reclaim the specified number of bytes by deallocating unneeded blocks.
Definition: SparseFile.cpp:160
SparseFileManager::ms_singleton
static boost::scoped_ptr< SparseFileManager > ms_singleton
Pointer to singleton.
Definition: SparseFile.h:498
SparseFile::CacheBlock
Definition: SparseFile.h:322
SparseFileManager::totalLoads
long long totalLoads()
Returns the total number of block loads in the cache.
Definition: SparseFile.cpp:292
SparseFileManager::resetCacheStatistics
void resetCacheStatistics()
Resets block load.
Definition: SparseFile.cpp:412