A Problem of Data Processing

Computer memory is limited. When the data size is large enough, the applications need the secondary storage such as hard disks to store the data.

An Example
If the data size is small, the data can be simply stored in memory. For example, the following program is compiled and executes successfully:


Array Size Is Fine.                 

#include  <iostream>  using namespace std;
int  main( ) {
  const   long  arraySize = 10000;
  double  array[arraySize];
  array[arraySize-1] = 10.0;
  cout << "The array size is fine.";
}
The Problem
If the memory is not able to hold all the data, the traditional data structures can not be used because they assume all data is in the memory. The following program does not pass the compilation because of not enough memory:


Array Size Is too Large.                 

#include  <iostream>  using namespace std;
int  main( ) {
  const   long  arraySize = 1000000000;
  double  array[arraySize];
  array[arraySize-1] = 10.0;
  cout << "The array size is fine.";
}
Solutions
There are two ways to solve the previous problem by storing the data in files or databases. Partial data is moved to the memory when requested.