How to work with large datasets in Replit

Before processing any large dataset, check your plan's memory allocation. The free Starter plan provides 2 GiB of RAM, Core gives 8 GiB, and Pro offers 8 GiB or more. You can see real-time usage by clicking the stacked computers icon on the left sidebar to open the Resources panel. This shows current RAM, CPU, and storage usage. If your app hits the memory ceiling, Replit kills the process with a 'Your Repl ran out of memory' error. Planning your data processing strategy around these limits is essential.

Open the Tools dock and click on Database to enable the built-in PostgreSQL database. Replit creates the database automatically and injects connection details as environment variables: DATABASE_URL, PGHOST, PGUSER, PGPASSWORD, PGDATABASE, and PGPORT. The development database is free with 10 GiB of storage. Use Drizzle Studio, accessible from the Database tool, to visually browse tables, run SQL queries, and inspect data without writing code. For production, a separate database instance is created with usage-based billing.

1-- Create a table for your dataset
2CREATE TABLE IF NOT EXISTS products (
3  id SERIAL PRIMARY KEY,
4  name VARCHAR(255) NOT NULL,
5  category VARCHAR(100),
6  price DECIMAL(10, 2),
7  created_at TIMESTAMP DEFAULT NOW()
8);
9
10-- Create an index for faster queries on common filters
11CREATE INDEX idx_products_category ON products(category);

Never use SELECT * FROM large_table without a LIMIT clause. Loading tens of thousands of rows into memory at once can crash your app. Instead, use LIMIT and OFFSET to fetch data in pages. For APIs, accept page and pageSize query parameters and calculate the offset. For batch processing, iterate through pages in a loop until no more rows are returned. This keeps memory usage constant regardless of total table size.

1import pg from 'pg';
2
3const pool = new pg.Pool({
4  connectionString: process.env.DATABASE_URL,
5});
6
7async function getProductsPage(page = 1, pageSize = 100) {
8  const offset = (page - 1) * pageSize;
9  const result = await pool.query(
10    'SELECT * FROM products ORDER BY id LIMIT $1 OFFSET $2',
11    [pageSize, offset]
12  );
13  return result.rows;
14}
15
16// Process all products in chunks
17async function processAllProducts() {
18  let page = 1;
19  const pageSize = 500;
20  let batch;
21
22  do {
23    batch = await getProductsPage(page, pageSize);
24    for (const product of batch) {
25      // Process each product without holding all in memory
26      await processProduct(product);
27    }
28    page++;
29  } while (batch.length === pageSize);
30}

When working with CSV files, JSON dumps, or other large files, use streaming instead of reading the entire file with fs.readFileSync or equivalent. Node.js streams process data line by line or chunk by chunk, keeping memory usage proportional to the chunk size rather than the file size. For CSV files, use a streaming parser like csv-parse. For JSON, use stream-json or process the file line by line if it contains one JSON object per line.

1import { createReadStream } from 'fs';
2import { parse } from 'csv-parse';
3
4async function importCSV(filePath) {
5  const parser = createReadStream(filePath).pipe(
6    parse({ columns: true, skip_empty_lines: true })
7  );
8
9  let count = 0;
10  let batch = [];
11
12  for await (const record of parser) {
13    batch.push(record);
14    count++;
15
16    // Insert in batches of 100 to reduce database round trips
17    if (batch.length >= 100) {
18      await insertBatch(batch);
19      batch = [];
20    }
21  }
22
23  // Insert remaining records
24  if (batch.length > 0) {
25    await insertBatch(batch);
26  }
27
28  console.log(`Imported ${count} records`);
29}

The Replit filesystem resets every time you deploy, so any files saved to disk during runtime are lost. For persistent file storage, use Replit Object Storage, which is backed by Google Cloud Storage. Object Storage handles images, videos, documents, and any unstructured data. Access it through the Tools dock. Files stored in Object Storage persist across deployments and restarts. Use it for user uploads, generated reports, or any file your app needs to keep permanently.

Track memory usage programmatically during heavy data processing to catch problems before they crash your app. In Node.js, use process.memoryUsage() to log heap statistics. Set a safety threshold below your plan limit and pause or abort operations when usage gets too high. Combine this with the visual Resources panel for real-time monitoring. If you consistently hit memory limits, consider upgrading your plan, reducing batch sizes, or offloading processing to a scheduled deployment that runs independently.

1function logMemoryUsage(label) {
2  const usage = process.memoryUsage();
3  const mbUsed = Math.round(usage.heapUsed / 1024 / 1024);
4  const mbTotal = Math.round(usage.heapTotal / 1024 / 1024);
5  console.log(`[${label}] Heap: ${mbUsed}MB / ${mbTotal}MB`);
6}
7
8// Use during batch processing
9async function processWithMonitoring() {
10  const MAX_HEAP_MB = 1500; // Safety threshold for 2 GiB plan
11  let page = 1;
12
13  while (true) {
14    const batch = await getProductsPage(page, 500);
15    if (batch.length === 0) break;
16
17    await processBatch(batch);
18    logMemoryUsage(`Page ${page}`);
19
20    const heapMB = process.memoryUsage().heapUsed / 1024 / 1024;
21    if (heapMB > MAX_HEAP_MB) {
22      console.warn('Memory threshold reached, pausing...');
23      if (global.gc) global.gc(); // Force GC if --expose-gc flag is set
24    }
25
26    page++;
27  }
28}