Tag Archives: Fast CSV Processing

Fast Displaying Large CSV File, Filter And Save Filtered Data Using SQLite & Java

SQLite CSV file often used for download or upload data from/to database. If table’s records count is large or very large then downloaded data to CSV file will have large in size.
Opening that large CSV file using spreadsheet application requires a lot of memory and CPU resources. Depending on user computer specifications, some consume much time to open.

It is much faster using Notepad++ to open large CSV file but if we want to filter data and save it to another CSV then using Notepad++ will not be efficient especially for some complex filtering criteria.

==> The workaround of this situation is to import that large CSV to SQLite. After importing to SQLite, we can filter it using SQL Query and save it back to CSV file.

According to documentation, SQLite is an embedded SQL database engine. SQLite reads and writes directly to ordinary disk files. A complete SQL database with multiple tables, indices, triggers, and views, is contained in a single disk file.
Think of SQLite not as a replacement for Oracle but as a replacement for fopen(). Its performance is usually quite good even in low-memory environments.

This blog post gives example on basic connection to SQLite, Sql Query, importing CSV, save query to CSV. Also example on using SQLite in Java so you can build your own application.
Continue reading

Advertisements