Hard Go with Large Spreadsheets
|
| Author | Content |
|---|---|
| robT Feb 08, 2011 12:11 PM EST |
I used LibreOffice spreadsheets to manipulate some large data sets (approx 8000 rows by 29 columns). What I did was use a bunch of vlookups to match dates between two data sets. LibreOffice crashed several times, took 100% of one dual core CPU and consumed 40-50% of 2GB of RAM. The saved spreadsheet was around 10MB. When I broke the problem up between two spreadsheets, CPU use dropped, but memory use went up to 90-100% of RAM (obviously swap was being used and performance dropped accordingly). |
| herzeleid Feb 08, 2011 12:19 PM EST |
Aside from the observation that this job seems to be perfectly suited to perl, what different spreadsheets have you tried with this? BTW this discussion brings to mind an office suite called "Applix" that I ran on linux in the 90s - it easily handled large data sets that would crash ms office every time, but unfortunately they are no longer in the market. |
| jacog Feb 09, 2011 3:41 AM EST |
Hey, I remember Applixware. Quite a decent suite, as I recall. And it's still around: http://www.vistasource.com/en/products.php But not too sure how it is these days. :) |
| herzeleid Feb 09, 2011 12:55 PM EST |
@jacog - thanks for the pointer, who knew? I actually did my first white paper with the current employer back in the 90s, on a solaris-2.5/sparc desktop, running applixware from a remote linux box. Good times... Edit: Checked out applixware at the link you gave - unfortunately it seems to be stagnant, stuck in a weird time warp when solaris/sparc and windoze peecee were the popular platforms... |
| Steven_Rosenber Feb 09, 2011 2:39 PM EST |
I'll be working on a 40,000-line spreadsheet soon, and I will report back on how that goes. |
| bigg Feb 09, 2011 3:29 PM EST |
Use R, not a spreadsheet, if you have 40,000 lines. |
Post a reply to this thread:
A free account is needed to post. Write your comment here first: your text is kept in this browser while you sign up or log in, and is waiting for you when you come back.
Your message:Markdown: **bold**, *italic*, [text](https://link), - lists, > quotes, $x^2$ for math. Formatting help
Sign up and post (it is free) or Log in and post
