[New Feature] [Beta] CSV Upload Overhaul

These are some good changes. Two questions:

Question 1:
Can I check the progress of an upload to plan for another upload or diagnose issues?

Question 2:
If it fails at a certain row (Say row 900), can it skip the row and continue on?

1 Like

Hi @josh10

  1. While the upload is in progress, you will see a progress bar and an ETA for when the upload will finish. You can also look at the data tab and see the data come in in real-time.
  2. Not with the current version

Hmm. I wonder if this is why uploads were getting a mysterious bug for me the past day or so.

Thank youuu!!! This had become in a pain for me limiting uploads to 500 lines! Hopefully this works faster!! Looking forward to this!

Thanks @cal @allenyang for working hard to develop this important upgrade!

I was part of the alpha testing an can attest that to uploading over 1 million records on multiple occasions. The csv file took under a minute to upload (depending on your speed) and the ETA was on average 1.1 days for the server to process the million rows. You can close your app or continue working on it while the backend processes your csv file. You’ll receive an e-mail once the process is complete. We can now upload millions of records in a reasonable amount of time!

7 Likes

How does this affect runtime uploads and do the old limits apply or can we exceed the plan limits now?

1 Like

What a great news!
I hope this will be implemented soon, it will really make uploading big files a lot easier!

If it fails at a certain row (Say row 900), can it skip the row and continue on?

Thanks to all!
Carlo

Hey team - I keep getting this error message. Is this related to the update?

Hi @james4,

I just pushed a fix to a seemingly similar bug - would you mind trying again and see if it works now? If it continues to fail, please submit a bug report so I can take a closer look.

1 Like

Works now, thanks for the quick fix

I’m getting this error when trying to upload to the Live db, but it was working fine on the dev side just last night. Help?

The .csv file is downloaded from google sheets, which encodes to UTF-8 automatically iirc

@cal some help when you get a minute :slight_smile:

Hi @andyestridge. If the file is in UTF-8 but is giving you that error, then it is probably a bug. Could you submit a bug report, sending in the CSV file you are having issues with, so I can take a look at it? Thanks!

1 Like

Just sent it in. Thanks!

Any chance you’ve looked at it? Sorry to bug you, it’s just that i had it all tested last night and was planning on applying things to the live app today :smiley:

Hi @andyestridge,

From investigating logs I can see that it triggered the ‘invalid UTF-8 file’ because there was a lone invalid character far down into your file. I’ve increased the intelligence of this check, so it should allow for things like that (because they do crop up in valid data), and the change should be going live later today. As a remedy if you need the upload before then you can find + remove that character from your file.

Ok thanks! And thanks for explaining why it was failing, that’s a big help.

:clap::clap::clap:

If you could focus on ignoring lines who give error codes instead of stopping the whole upload, would be a great! Thanks

could you advise on what characters are not allowed? When i run my file through a UTF-8 validator it comes back clean…